Knowledge discovery system

ABSTRACT

A knowledge discovery system configured to present global and local context sensitive and efficient augmented content in accordance with multiple criteria, including one or more user&#39;s preferences. The system enables one or more registered users to allocate, generate and share augmented content with other registered or unregistered users based on a sharing policy for each user. The system provides a first user the ability to generate augmented content using at least one of the first user&#39;s preferences, manual annotation, and profile; and automatically allocate, regenerate, or modify the augmented content generated for the first user using at least one of a designated process, preconfigured preferences of a designated process, preferences of a second user, a profile of a second user, preconfigured preferences of a registered user, and/or preconfigured preferences for an unregistered user. Real-time and theme based content augmentation may also be used to further enhance the user&#39;s experience.

PRIORITY CLAIM

This application claims priority to, and is a continuation in part of,U.S. application Ser. No. 14/217,462 filed on Mar. 17, 2014, entitled“System and method for augmented knowledge discovery” which claimspriority to U.S. Provisional Patent Application having application No.61/801,359, filed Mar. 15, 2013, and is a continuation in part of U.S.application Ser. No. 14/491,977 filed on Sep. 19, 2014, entitled “Systemfor Knowledge Discovery Platform” which claims priority to U.S.Provisional Patent Application having application No. 61/880,175, filedSep. 19, 2013, and is a continuation in part of U.S. application Ser.No. 15/057,052 filed on Feb. 29, 2016, entitled “System for knowledgediscovery” which is a continuation of and claims priority to U.S.application Ser. No. 13/573,564 filed on Sep. 24, 2012 which claimspriority to U.S. Provisional Patent Application having application No.61/743,047, filed Aug. 24, 2012 and to U.S. Provisional PatentApplication having application No. 61/626,253, filed Sep. 23, 2011. Eachof the above named applications is hereby incorporated by referenceherein in its entirety.

TECHNICAL FIELD

The subject of this application generally relates to informationprocessing and knowledge discovery, and data presentation particularlyto systems and methods for improving data-exploration, learning andbrowsing and, and more specifically for context sensitive dataaugmentation for a richer user experience, data-exploration, knowledgediscovery, and learning systems.

BACKGROUND

Having a context sensitive user interface which can automatically choosefrom a multiplicity of options based on the current or previous state(s)of a program operation can be found in current graphical user interface.For example: Clicking on a text document automatically opens thedocument in a word processing environment. The user does not have tospecify what type of program to use to open the file. Program files andtheir shortcuts (i.e. executable files) can be associated with certaintype of files, e.g. text document, and are automatically run by theoperating system when the user selects or double clicks the file.Similarly, the user-interface may also provide context sensitivefeedback, such as changing the appearance and/or color of the mousepointer or cursor. In addition, context sensitive feedback may also beused in video games where it change a button's function based on aplayer who is in a certain position or a place and needs to interactwith an object.

Relational databases are currently the predominant choice in storingdata like financial records, medical records, personal information,manufacturing and logistical data. Nowadays large-scale data orinformation processing can involve various types of collection,extraction, warehousing, analysis and statistics. For example,organizing and matching data by using some common characteristics foundwithin the data set would result in new groups of data that canorganized and are easier for many people to understand, search, indexand manipulate.

By describing the contents and context of data files, the quality ofprocessing the original data files can be greatly increased. Forexample, a webpage may include metadata specifying what language wasused in writing its code, what tools were used to create it, and whereto go for more on the subject, higher-level concepts that describe thedata. Thus allowing browsers to automatically improve the experience ofusers. The results of any large-scale data processing can be anextensive set of meta-data, data, and relationships that may be used ina search engine, for example, to provide a possible set of relatedinformation to a term that is used in a search query. For example,search engines have used and generated enormous amount of data andmetadata that is used to provide links to content that may be ofpossible interest to a user based on what the user is searching for.

As stored digital information has increased tremendously in size, theability of a user to use effectively personal data, corporate data, orpublically available data has also increased many folds although itstill falls short of the potential of reasoning about the large amountof data that is available and continues to grow at an astounding pace.Therefore, there exists a need to more effectively use and reason aboutthe data, and with more a richer augmented user experience whilereading, writing, searching, or using digital data information.

Large amount of data can be stored using various types of relationaldatabases, network based storage, or cloud based storage. These are butsome examples of predominant choices in storing data and informationlike financial records, medical records, personal information,manufacturing and logistical data. Nowadays large scale data orinformation processing can involve various types of collection,extraction, warehousing, analysis and statistics. For example,organizing and matching data by using some common characteristics foundwithin the data set would result in a new groups of data that can beorganized and are easier, for many people, to understand, search, indexand manipulate.

As stored digital information has increased tremendously in terms ofsize or amount of data information, the ability of a user to useeffectively personal data, corporate data, or publically available datahas also increased many folds. Additional problems are encountered infinding relevant data for a user's needs. Knowledge discovery platform,systems as described in related patent applications can be used togenerate augmented knowledge using such large scale data that meet theneeds of a user. The augmented knowledge provided to the user can behighly relevant to another user or another knowledge discovery system.However, the other user or system may have certain distinct criterions,characteristics, preferences or interests that are different from thefirst user.

Thus, an increase in accuracy and efficiency can be achieved bybenefiting from using the augmented knowledge already obtained for afirst user and by a regenerated or modified augmented content andknowledge tailored to a second user's interest, profile, or preferences.Therefore, there exists a need for a knowledge discovery system that canleverage the knowledge discovered for a first user to provide augmentedknowledge and/or newly discovered or augmented knowledge based on asecond user's preferences or interests.

SUMMARY

This disclosure presents new and useful methods and systems to providemultilevel context sensitive augmented experience, browsing, dataexploration, knowledge discovery, and e-learning. In accordance with oneembodiment, this multilevel context sensitive augmented content ispresented using overlaid layers on top of the digital information(reference content or original content) being viewed by a user.Furthermore, the overlaid layers can be transparent or translucent for anon-obtrusive user experience. Thus, providing the user the ability tointeract with the original content while viewing a dynamically updatedaugmented content on top of the original content, the updated augmentedcontent is generated based at least on the user interaction with thereference content. Furthermore the user can manipulate the originalcontent and its associated or related categories and other relevantaugmentation data to generate more relevant and meaningful augmentationwhile viewing the augmented content on top of the reference content.

In accordance with one embodiment, a system for generating andpresenting augmented content on a translucent display layer overlaid ontop of a reference content display layer on the same display screen. Theaugmented content is generated using relevant features of the referencecontent or the displayed portion of the reference content. Thegeneration of the augmented content is further customized usinguser-relevant characteristics, attributes, history, and relevantfeatures in relation to the reference content such as generic categoriesand relationships. In addition, the user controls the position and sizeof both the reference content display layer and the augmented contentdisplay layers on the same display screen, as well as the ability of theuser to control the visibility and hiding of all display layers.Furthermore, the user controls the sharing of the same display screen bythe reference content and augmented content display layers.

In accordance with one embodiment, the system for generating andpresenting augmented content provides a set of augmentation filters:topics and categories based on the reference content to aid the user infurther customization of the augmentation filters to suite his/herinterests. The generated augmentation content is one or more of onlinedocuments, web pages, web links. The generated augmentation content canbe a customized version using a variety of ways such as presenting asummary of the augmented content, or in deleting un-necessary links andads. In accordance with one embodiment, the generated augmentationcontent is based at least on one of (i) a set of criteria associatedwith the reference content, (ii) user customization of augmentationfilters, (iii) user interaction with the reference content, and (iv)user interaction with the generated augmented content.

In accordance with one embodiment, the system can employ the samemethods and algorithms to enable the user to custom build a knowledgegraph of concepts and relationships based on information retrieved fromstructured and unstructured data residing in a private or public datastore or other public repositories. The Augmentation System relies onthese data sources along with the user's feedback and interests togenerate on the fly relevant augmentation data for the task at hand. Forexample, a physician can utilize this system to custom build a knowledgegraph for a patient based on the physician's experience and knowledge,the patient's history, the patient's known diseases, symptoms, andailments, and known public data related to the patient's case. Such asystem will enable the physician to make educated and informed decisionsinstead of being mired in a plethora of sources where it would beextremely hard for the physician to manually extract reliable andrelevant data in an efficient and useful way.

In accordance with one embodiment, the system for generating andpresenting augmented content dynamically updates the augmented contentby utilizing additional filters, metrics, and customization provided bythe user as a result of the generated augmented content. Furthermore,the user can save any or all the data associated with a particularsession of data augmentation. This will enable the user to build on theaugmentation of previous sessions.

In accordance with one embodiment, the system for generating andpresenting augmented content generates global and local augmentationcontent associated with the reference content and any selected orhighlighted part of it. For example, the system generates a plurality ofglobal augmentation content based on the augmentation filters associatedwith the overall reference content, and the system generates a pluralityof local augmentation content based on a specific part of the referencecontent that is selected or flagged by the user, or currently beingviewed by the user.

In accordance with one embodiment, the system for generating andpresenting augmented content enables collaborative augmentation, e.g. auser can share the generated augmented content with other users.Furthermore, the user can share the content augmentation filters or thesettings used to generate the augmented content with other users.

In accordance with one embodiment, the system for generating andpresenting augmented content enables a user to make use of nestedhierarchical content augmentation capabilities. A user can requestcontent augmentation using at least a portion of a previously generatedaugmented content. The previously generated augmented content serves asnew reference content for the system to generate and present to the usera new augmented content. The user can traverse the content augmentationgraph to further customize the content augmentation at any level.

In accordance with one embodiment, the display screen may be physicallyattached to an electronic device, e.g. a mobile device, a handhelddevice, a tablet, etc . . . , or the display screen may physicallyseparate from the electronic device. For example a touch display where auser interacts with the display screen and controls both the positionand size of the various display layers on the display screen. Thedisplay screen can communicate with a remote electronic device such as aremote server, or a mobile device. Alternately, the user can control theposition and size of all display layers on the physically detacheddisplay screen using the electronic device.

In accordance with one embodiment, the user interaction with thereference content includes at least one of a manipulation of a region ofthe first display layer, a manipulation of a region of the seconddisplay layer, hiding of the first display layer, hiding of the seconddisplay layer, saving the first set of augmented content, saving aportion of the first set of augmented content, modifying thetranslucency of the second display layer, a selection of a region of thefirst display screen, a manipulation of a region of the first displayscreen, one or more user gesture made onto the first display screen, anactivation of a button of the first display screen, an activation of abutton of the electronic device, and using a human interface device tocommunicate the user interaction to the electronic device.

In accordance with one embodiment, the reference content, local content,augmentation content are displayed using multiple display layers bymeans of one or more display screens. The display screen compriseselectronic system to receive and/or transmit information to anelectronic device. The user interaction with the reference contentincludes the manipulation of one or more regions of at least one displaylayer, a manipulation of one or more regions of at least one displaylayer of the augmented content, hiding of any one or more of the displaylayers, saving the first set of augmented content, saving a portion ofthe first set of augmented content, modifying the translucency of anyone of the display layers, a selection or a manipulation of one or moreregions of any one of the display screens, one or more user gesture madeonto the display screen, an activation of a button of the displayscreen, an activation of a button of the electronic device, and using ahuman interface device to communicate the user interaction to theelectronic device or to the display screen.

In accordance with one embodiment, this disclosure refers to augmentinga given content based on a number of manually defined and automaticallyextracted parameters to generate a set of local and global dataelements. The set of local and global data elements can be used in avariety of application specific augmentation systems to enhance a user'sexperience while interacting with the given content.

In accordance with one embodiment, this disclosure facilitates theconstruction and presentation of a user-customized network of concepts,objects and relationships that serve to augment the content at hand forthe purpose of knowledge discovery, learning, and a richer userexperience in browsing and/or interacting with data information.Furthermore, the constructed network can be saved and further augmentedover time for richer and more efficient user experience. This is incontrast to having a pre-built network of concepts and relationshipsthat a user can access. This system generates a network that can becustomized and tailored based on the user's interests.

In accordance with one embodiment, this disclosure facilitates a systemthat provides the user the ability to fully control the generatedaugmented content by virtue of changing the scope of certain topics,e.g. expanding or specifying a narrower sub-topic, based at least on oneof a defined theme, predefined themes, and categories. Therefore, theaugmented content can serve to further explain, define, and to elaborateand expound on reference content or a selected portion of referencecontent being viewed, observed, or interacted with by a user.

In accordance with one embodiment, this disclosure can be used toaggregate information related to a reference or selected content bycustomizing the augmentation filters to achieve the desired or intendedresults. For example, the information, reference content, or thegenerated augmented content can include rich media like video, audio,images as well as text. Various filters can be customized by the user toenable a user to increase the relevance of the generated augmentedcontent to the intended user objective. In addition, a hierarchicalsystem of content augmentation maybe defined and customized by aselected theme or a category. The generated augmented content and itsdisplay layers can be monetized for ads and other monetization purposes.

In accordance with one embodiment, this disclosure enables real-timemanipulation of reference and augmented content for enhanced and richerUser Experience (UX). In addition, collaboration and sharing ofaugmented content provides an increase in value and productivity to auser. Similarly, collaboration and sharing of augmentations filters andsettings provide additional richness and ease of viewing, browsing,sharing, and manipulation of reference and augmented content.Furthermore, the user is able to control the presentation style of thegenerated augmented content, e.g. as raw links, concise summary ofaugmented content, or other methods that capture the essence of theaugmented content. The presentation style of the generated augmentedcontent maybe for data analysis, research, information, monetization,commercial, or educational purposes.

In accordance with one embodiment, a system for generating andpresenting augmented content on a translucent display layer overlaid ontop of a reference content display layer on the same display screen. Theaugmented content is generated using relevant features and filtersextracted from the reference content or the displayed portion of thereference content. A feature is a pattern that can be extracted orinferred from the content at hand. Feature extraction is the process ofreducing the dimensionality of a document by capturing a set of featureswhich reflect the most relevant and salient properties of that document.For example, a feature can be a keyword in the content, title of thecontent, or other metadata that can be extracted or inferred from thecontent, its link, or any embedded content or link to other content. Afeature could also correspond to a concept such as a name, a topic, oran event that can be extracted or inferred from the content. A group offeatures are combined together using an association rule to form apattern, a complex pattern, or a filter. A filter may comprise ordescribe a relationship between features, a collection of features, or agroup of features. A filter may also reflect a correlation between a setof features. A category is a grouping of features or a grouping ofmultiple sets of features. A category may correspond to a classificationof entities or concepts that share some property or relationships. Acategory maybe formed using a filter, group of filters, or anycombination of filters and features. An association rule to combine afeature, a set of features, a filter, or a set of filters can also beused to generate a category, a set of categories, a new feature, a newfilter, a new set of features, or a new set of filters. Thus, thegeneration of the augmented content can use (i) any one of a feature, afilter, a category, or (ii) any association in between, or acombination, of a filter, a feature, and a category. The generation ofaugmented content is further customized using user-relevantcharacteristics, attributes, history, and other relevant features inrelation to the reference content such as generic categories andrelationships. In addition, the user controls the position and size ofboth the reference content display layer and the augmented contentdisplay layers on the same display screen, as well as the ability of theuser to control the visibility and hiding of all display layers.Furthermore, the user controls the sharing of the same display screen bythe reference content and augmented content display layers.

In accordance with one embodiment, the system for generating andpresenting augmented content provides a set of augmentation filters:topics and categories based on the reference content to aid the user infurther customization of the augmentation filters to suite his/herinterests. The generated augmentation content is one or more of onlinedocuments, web pages, web links. The generated augmentation content canbe a customized version using a variety of ways such as presenting asummary of the augmented content, or in deleting un-necessary links andads. In accordance with one embodiment, the generated augmentationcontent is based at least on one of (i) a set of criteria associatedwith the reference content, (ii) user customization of augmentationfilters, (iii) user interaction with the reference content, and (iv)user interaction with the generated augmented content.

In accordance with one embodiment, the system can employ the samemethods and algorithms to enable the user to custom build a knowledgegraph of concepts and relationships based on information retrieved fromstructured and unstructured data residing in a private or public datastore or other public repositories. The Augmentation System relies onthese data sources along with the user's feedback and interests togenerate on the fly relevant augmentation data for the task at hand. Forexample, a physician can utilize this system to custom build a knowledgegraph for a patient based on the physician's experience and knowledge,the patient's history, the patient's known diseases, symptoms, andailments, and known public data related to the patient's case. Such asystem will enable the physician to make educated and informed decisionsinstead of being mired in a plethora of sources where it would beextremely hard for the physician to manually extract reliable andrelevant data in an efficient and useful way.

In accordance with one embodiment, the system for generating andpresenting augmented content dynamically updates the augmented contentby utilizing additional filters, metrics, and customization provided bythe user as a result of the generated augmented content. Furthermore,the user can save any or all the data associated with a particularsession of data augmentation. This will enable the user to build on theaugmentation of previous sessions.

In accordance with one embodiment, the system for generating andpresenting augmented content generates global and local augmentationcontent associated with the reference content and any selected orhighlighted part of it. For example, the system generates a plurality ofglobal augmentation content based on the augmentation filters associatedwith the overall reference content, and the system generates a pluralityof local augmentation content based on a specific part of the referencecontent that is selected or flagged by the user, or currently beingviewed by the user.

In accordance with one embodiment, the system for generating andpresenting augmented content enables collaborative augmentation, e.g. auser can share the generated augmented content with other users.Furthermore, the user can share the content augmentation filters or thesettings used to generate the augmented content with other users.

In accordance with one embodiment, the system for generating andpresenting augmented content enables a user to make use of nestedhierarchical content augmentation capabilities. A user can requestcontent augmentation using at least a portion of a previously generatedaugmented content. The previously generated augmented content serves asnew reference content for the system to generate and present to the usera new augmented content. The user can traverse the content augmentationgraph to further customize the content augmentation at any level.

In accordance with one embodiment, the display screen may be physicallyattached to an electronic device, e.g. a mobile device, a handhelddevice, a tablet, etc . . . , or the display screen may physicallyseparate from the electronic device. For example a touch display where auser interacts with the display screen and controls both the positionand size of the various display layers on the display screen. Thedisplay screen can communicate with a remote electronic device such as aremote server, or a mobile device. Alternately, the user can control theposition and size of all display layers on the physically detacheddisplay screen using the electronic device.

In accordance with one embodiment, the user interaction with any one ofthe reference content, a portion of the reference content, an augmentedcontent, and a portion of an augmented content includes at least one ofa manipulation of a region of the first display layer, a manipulation ofa region of the second display layer, hiding of the first display layer,hiding of the second display layer, hiding a display layer, saving atleast a portion of the reference content, saving at least a portion ofthe augmented content, saving at least a portion of one or more of a setof filters, a set of features, a set of categories, a set of metrics, aset of user preferences, modifying the translucency of a display layer,a selection of displayed content using a region of the first displayscreen, a manipulation of displayed content using a region of the firstdisplay screen, one or more user gesture made onto the first displayscreen, an activation of a button of the first display screen, a userinput made using the electronic device, and using a human interfacedevice to communicate the user interaction to the electronic device.

In accordance with one embodiment, the reference content, local content,augmentation content are displayed using multiple display layers bymeans of one or more display screens. The display screen compriseselectronic system to receive and/or transmit information to anelectronic device. The user interaction with the reference contentincludes the manipulation of one or more regions of at least one displaylayer, a manipulation of one or more regions of at least one displaylayer of the augmented content, hiding of any one or more of the displaylayers, saving the first set of augmented content, saving a portion ofthe first set of augmented content, modifying the translucency of anyone of the display layers, a selection or a manipulation of one or moreregions of any one of the display screens, one or more user gesture madeonto the display screen, an activation of a button of the displayscreen, an activation of a button of the electronic device, and using ahuman interface device to communicate the user interaction to theelectronic device or to the display screen.

In accordance with one embodiment, this disclosure refers to augmentinga given content based on a number of manually defined and automaticallyextracted parameters to generate a set of local and global dataelements. The set of local and global data elements can be used in avariety of application specific augmentation systems to enhance a user'sexperience while interacting with the given content.

In accordance with one embodiment, this disclosure facilitates theconstruction and presentation of a user-customized network of concepts,objects and relationships that serve to augment the content at hand forthe purpose of knowledge discovery, learning, and a richer userexperience in browsing and/or interacting with data information.Furthermore, the constructed network can be saved and further augmentedover time for richer and more efficient user experience. This is incontrast to having a pre-built network of concepts and relationshipsthat a user can access. This system generates a network that can becustomized and tailored based on the user's interests.

In accordance with one embodiment, this disclosure facilitates a systemthat provides the user the ability to fully control the generatedaugmented content by virtue of changing the scope of certain topics,e.g. expanding or specifying a narrower sub-topic, based at least on oneof a defined theme, predefined themes, and categories. Therefore, theaugmented content can serve to further explain, define, and to elaborateand expound on reference content or a selected portion of referencecontent being viewed, observed, or interacted with by a user.

In accordance with one embodiment, this disclosure can be used toaggregate information related to a reference or selected content bycustomizing the augmentation filters to achieve the desired or intendedresults. For example, the information, reference content, or thegenerated augmented content can include rich media like video, audio,images as well as text. Various filters can be customized by the user toenable a user to increase the relevance of the generated augmentedcontent to the intended user objective. In addition, a hierarchicalsystem of content augmentation maybe defined and customized by aselected theme or a category. The generated augmented content and itsdisplay layers can be monetized for ads and other monetization purposes.

In accordance with one embodiment, this disclosure enables real-timemanipulation of reference and augmented content for enhanced and richerUser Experience (UX). In addition, collaboration and sharing ofaugmented content provides an increase in value and productivity to auser. Similarly, collaboration and sharing of augmentations filters andsettings provide additional richness and ease of viewing, browsing,sharing, and manipulation of reference and augmented content.Furthermore, the user is able to control the presentation style of thegenerated augmented content, e.g. as raw links, concise summary ofaugmented content, or other methods that capture the essence of theaugmented content. The presentation style of the generated augmentedcontent maybe for data analysis, research, information, monetization,commercial, or educational purposes.

Described herein is a system and method for performing knowledgediscovery in a computer system having memory including coupling thememory to a processor via a memory channel, wherein the computer systemis operable to have independent access to the memory channel and to astorage medium where the knowledge discovered representing augmenteddata is stored. Knowledge Discovery Platform (KDP) system is describedto enable one or more users of KDP system to allocate and shareaugmented content or discovered knowledge with one or more end/otherusers using the computer system, multiple computer systems that canaccess the storage medium where the knowledge discovered or augmenteddata are stored, or via any one or more of a plurality of means forcommunication wired or wirelessly including devices for accessing suchcommunication systems, means for notification, and/or through a processusing a KDP system. The one or more users of a KDP system can invokeknowledge discovery request or data augmentation request on certain usercontent. The user content may be displayed using a wireless device or amonitor coupled to the computer system. The KDP system automaticallygenerates augmented content and display the augmented data based onuser's parameters, preferences, or interests. The one or more users cantailor or annotate the augmented content in accordance with certainparameters, preferences, or interests.

Also described herein is a method for performing data augmentationrequest in a computer system for providing enriched or augmented sharingof the knowledge discovered (or already augmented user content) betweenone or more users of KDP system and with one or more end users (clustersof users), another KDP system, or a process using another KDP system. Areceiving end user or process would receive the augmented contentinformation or a notification pointing to the augmented content wherethe augmented content can be accessed, downloaded or manipulated by theend user or process using the KDP system.

Also described herein is a method for performing data augmentationrequest in a computer system for providing the receiving end user (or aprocess using another instance of a KDP system) may perform automatedprocessing of the received notification of the augmented content usingcertain preferences, preconfigured preferences, or having certainprogrammable parameters such that the KDP system regenerates the sharedaugmented content using the preferences, preconfigured preferences, orcertain programmable parameters of the process. The regeneratedaugmented content can be parsed or additional invocation of the KDPsystem may be used to further refine the augmented content or forsharing a customized version of the received augmented content. Theaugmented content can be stored in the computer system or transmitted tobe processed further through additional computer systems, KDP systems,or using other computer systems or processes that are dedicated forprocessing knowledge discovery request or data augmentation request inresponse to one or more preferences, specific interests, and/or targetmarket. Real-time and theme based augmentation may also be used tofurther enhance the user's experience.

In accordance with one embodiment, the present application disclosesknowledge discovery platform systems and methods to provide a first userthe ability to generate augmented content, and to allocate, regenerate,or modify the augmented content using stored information in a computersystem, and the stored information is associated with the first user.Furthermore, the stored information includes preferences or otherprogrammable parameters associated with a second user, a cluster ofusers, or augmented content of the first user. The stored informationmay also include any one of a profile of a second user, a parameter ofan executable code or process, preconfigured preferences for aregistered KDP user, and preconfigured preferences for an unregisteredKDP user. The computer system is programmable to collect the storedinformation using a temporary (volatile memory) or permanent storage(non-volatile memory) of the profile of the second user, the parameterof the executable code or process, the preconfigured preferences forregistered KDP users, and preconfigured preferences for an unregisteredKDP user.

In accordance with one embodiment, the present application disclosesknowledge discovery platform systems and methods to store a KDP'sprofile of a user, knowledge discovery request, or data augmentationrequest and provide a first user the ability to generate augmentedcontent using new preferences and/or KDP's profile of the first user,and to allocate, regenerate, or modify the augmented content usingpreferences, parameters, KDP's profile, or information associated withthe augmented data or discovered knowledge of a second KDP user.

In accordance with one embodiment, the present application disclosesknowledge discovery platform systems and methods to (i) provide a firstuser the ability to generate augmented content using at least one of afirst user's preferences, first user's manual annotation, and firstuser's KDP profile; and (ii) automatically allocate, regenerate, ormodify the augmented content using at least one of a designated process,preconfigured preferences of a designated process, preferences of asecond user, a KDP's profile of a second user, preconfigured preferencesof a registered KDP user, and preconfigured preferences for anunregistered KDP user. The designated process can be a process or partof a process being executed using a KDP system, a computer system, acompute server or a wireless device.

In accordance with one embodiment, one or more users of KDP are able toallocate and share augmented content (or discovered knowledge) with oneor more end users using any one or more of means for communication,means for notification, and through a KDP process. The one or more usersof KDP invoke KDP system on certain content, and the KDP systemautomatically generates augmented content. The one or more users cantailor or annotate the augmented content in accordance with certainparameters, preferences, or interests. The one or more users then sharethis augmented content with one or more end users or processes. Areceiving end user or process would receive the augmented contentinformation or a notification pointing to the augmented content wherethe augmented content can be accessed, downloaded or manipulated by theend user or processed using the KDP system.

In accordance with one embodiment, a process may perform automatedprocessing of the received notification of the augmented content usingcertain preconfigured preferences, or programmable parameters, such thatthe KDP system regenerates the shared augmented content using thepreconfigured preferences or the programmable parameters of the process.The regenerated augmented content can be parsed or additional invocationof KDP system may be used to further refine the received augmentedcontent or for sharing a customized version of the received augmentedcontent. The received augmented content can be processed throughadditional systems or using other processes that are dedicated to one ormore preferences, specific interests, localized parameters, geographicallocation, and/or a target market.

In accordance with one embodiment, a user is enabled to share theaugmented content (or a particular knowledge graph or other generatedcontent) with one or more designated end users or processes inaccordance with one or more predefined service levels, customizedmarket, geographical locations, seasonal or timing events, and/orlocalized preferences per end user.

In accordance with one embodiment, if an end user is a registered KDPuser then certain privileges or service levels can be invoked orprocessed to enrich or further customize the augmented content accordingto the end user profile.

In accordance with one embodiment, if an end user is not a registeredKDP user then the KDP system further customizes the regeneration of thereceived augmented content by (i) restricting or limiting certainprivileges or service levels, (ii) enabling certain privileges orredirecting to certain service levels, or (iii) using certain localizedparameters that are associated with the end user.

In accordance with one embodiment, certain service levels or privilegesmay be invoked once the end user has become a registered KDP user. Aregistered KDP user can tailor and automate the knowledge discovery andits broadcasting to the masses. This would enable a registered KDP usera rich and unique offering that will incentivize the growth of thenumber of registered KDP users and enabling, for example, people todiscover stories or information, annotate them and share them with theirfriends or the public at large by sharing the story on any social orpublic network or forum, the KDP system in turn further customizes thedelivered or shared stories or information using any one of theembodiment disclosed above or any combination of one or more of theembodiments disclosed above.

BRIEF DESCRIPTION OF THE DRAWINGS

The attached drawings, which are incorporated into and constitute a partof this specification, illustrate one or more examples of embodiments.These drawings together with the description of example embodimentsserve to explain the principles and implementations of the embodiments.

FIG. 1 shows Discovery Patterns.

FIG. 2 shows block diagram for an Augmentation System.

FIG. 3 shows block diagram for Causality and Augmentation System.

FIG. 4 shows block diagram of a Causality Graph Synthesis.

FIG. 5 shows block diagram of the Augmentation System with the causalitygraph.

FIG. 6 is a block diagram of a data augmentation system 100 that is usedto generate augmented content for a given reference content.

FIG. 7 is a block diagram of a hierarchical augmentation system 200 thatis used to support the generation of multilevel augmented content usingmultiple reference content.

FIG. 8 is a block diagram of a hierarchical augmentation system 300 thatis used to support the generation of multilevel augmented content usingmultiple reference content along with a controller that manages thenested augmentation functions and the user's interaction with thegenerated augmented content.

FIG. 9 is a block diagram of a data augmentation system 400 having thecapability of manipulating, controlling and displaying both thereference content and the augmented content simultaneously anddynamically.

FIG. 10 is a block diagram of a relevant augmented content extraction500 that is used as a subsystem of a data augmentation system.

FIG. 11 is a block diagram of a relevant augmented content extraction600 that is used as a subsystem of a data augmentation system.

FIG. 12 is a display example of the generated augmented content of usingmultiple display layers.

FIG. 13 is a display example of a simulated use case of a dataaugmentation system invoked while viewing a news article.

FIG. 14 is a display example of a simulated use case of a userinteraction with a data augmentation system invoked while viewing a newsarticle.

FIG. 15 shows an example block diagram of knowledge discovery system inaccordance with one embodiment.

FIG. 16 shows an example block diagram of knowledge discovery system inaccordance with one embodiment.

FIG. 17 shows an example block diagram of Knowledge Discovery Tailoringand Annotation system in accordance with one embodiment.

FIG. 18 shows an example block diagram of Knowledge Sharing andBroadcasting in accordance with one embodiment.

DETAILED DESCRIPTION

The present disclosure presents techniques, systems and methods toprovide a user with global and local context sensitive augmented contentto enhance the user experience while interacting with digitalinformation be it while reading, writing, drawing, browsing, searching,viewing, or using digital data information such as financial, medical,business or corporate data, social media data, or any data that isaccessible locally or on the web and/or remotely through web basedservices. These techniques, systems and methods are applicable tovarious computing platforms such as hand-held devices, desktopcomputers, notebook computers, mobile devices, as well as computeservers.

The term “coupled” is defined as connected, although not necessarilydirectly, and not necessarily mechanically. The terms “a” and “an” aredefined as one or more unless this disclosure explicitly requiresotherwise. The terms “comprise” (and any form of comprise, such as“comprises” and “comprising”), “have” (and any form of have, such as“has” and “having”), “include” (and any form of include, such as“includes” and “including”) and “contain” (and any form of contain, suchas “contains” and “containing”) are open-ended linking verbs. As aresult, a method or device that “comprises,” “has,” “includes” or“contains” one or more steps or elements possesses those one or moresteps or elements, but is not limited to possessing only those one ormore elements. Likewise, a step of a method or an element of a devicethat “comprises,” “has,” “includes” or “contains” one or more featurespossesses those one or more features, but is not limited to possessingonly those one or more features. Furthermore, a device or structure thatis configured in a certain way is configured in at least that way, butmay also be configured in ways that are not listed.

Example embodiments are described herein in the context of a system ofone or more mobile device, electronic device, handheld device,computers, servers, firmware, and software. Those of ordinary skill inthe art will realize that the following description is illustrative onlyand is not intended to be in any way limiting. Other embodiments willreadily suggest themselves to such skilled persons having the benefit ofthis disclosure. Reference will now be made in detail to implementationsof the example embodiments as illustrated in the accompanying drawings.

In the interest of clarity, not all of the routine features of theimplementations described herein are shown and described. It will, ofcourse, be appreciated that in the development of any such actualimplementation, numerous implementation-specific andapplication-specific decisions must be made in order to achieve thedeveloper's specific goals, such as compliance with business-relatedconstraints, and that these specific goals will vary from oneimplementation to another and from one developer to another. Moreover,it will be appreciated that such a development effort might be complexand time-consuming, but would nevertheless be a routine undertaking ofengineering for those of ordinary skill in the art having the benefit ofthis disclosure.

In general, various information processing techniques and algorithms canbe used to provide the augmented data system with global and localcontext sensitive augmented content. In the following paragraphs certaindefinitions and representations of data flow models are presented anddiscussed without limitations on how each model may be implementedwhether by hardware, software, firmware, or any combination thereof.

Multilevel, e.g. global and local, context sensitive augmented contentwould increase productivity and enhance a user's experience whileviewing or interacting with data for the purpose of learning, reading,writing, drawing, browsing, searching, discovering, viewing images orany type of user interaction with digital data information whetherstructured or unstructured (e.g. financial, health, manufacturing, andcorporate data). The digital data information may be stored locally orremotely via a corporate server or in the cloud. Additionally private aswell as public sources of data may be used or selected by the user forthe ultimate personalized range of choices that may be used to furthernarrow down or expand the augmented content being presented.

Additionally, multilevel context sensitive augmented content wouldincrease productivity and enhance business intelligence for theenterprise by providing context sensitive augmented content that isgenerated by dynamically mining and analyzing structured andunstructured enterprise data and/or possibly leveraging structured andunstructured publically available data for further improving userexperience. In addition, multilevel context sensitive contentaugmentation filters provide the ability to dynamically mine data on thefly based on modification of a new input from a user. For example, a newinput from a user can be the selection of a new text or a portion of thereference content, or it can be a feedback provided such as elevatingthe priority or weight (e.g. like) or decreasing the priority or weight(e.g. dislike, delete, dismiss) a single augmented content, a categoryof augmented content, or a theme of augmented content. Furthermore,leveraging the history and/or user personal preferences, the multilevelcontext sensitive augmented content can be further in tune with what theuser would like to see or expects to see in the augmented content beinggenerated and presented.

In accordance with one embodiment, a feature of the multilevel contextsensitive augmented content is that the augmented content is generatedeither in the cloud or locally using sophisticated information retrievalalgorithms or using a set of heuristics so as to enable large-scale dataprocessing, information retrieval, and web mining. Knowing thatextracting a feature set from a web page is a problem that is known andvarious algorithms, methods, and research into various solutions havebeen made, this system can use existing research or methodologies toextract a feature set. Furthermore, this system employs a set ofheuristics and metrics that efficiently extract a set of features thatcharacterize the reference content at hand. These heuristics rely onembedded hints, metrics, meta-data, or other embedded knowledge andinformation that can be extracted from the structure, url link, embeddedlinks, title of the document, or other types of data that may bedirectly or indirectly related to the reference content along withfeedback provided by the user.

In accordance with one embodiment, a feature of the multilevel contextsensitive augmented content is that the information retrieved andknowledge constructed can be saved and called upon in futureaugmentation tasks and sessions.

In accordance with one embodiment, a feature of the multilevel contextsensitive augmented content is that the augmented content is presentedthrough a translucent layer on top of the original content being viewedby the user. Hence a non-obtrusive content augmentation that is hiddenor made available whenever a user disables or enables the global andlocal context sensitive augmented content application. Relevantaugmented content are displayed on top of a translucent layer on top ofthe original content being viewed by the user. Hence, the augmentationsystem provides a less obtrusive and more efficient interaction,browsing and exploration experience.

In accordance with one embodiment, a multilevel corresponds to at leasttwo levels, a global level and a local level. A global and a localrelevant features of reference content, maybe defined as a globalrelevant feature corresponding to a feature or a theme common throughoutthe reference content, and a local relevant feature corresponding to afeature strongly related to a locality within the reference content. Onemethod of dynamically updating augmented content can be achieved byleveraging real-time user feedback, such as elevating priority ordismissing augmented content as being presented to the user. If anaugmented content's priority is elevated, its weight increases as wellas the metadata that describes this augmented content gets promotedwhich in turn updates existing augmentation filters as well asgenerating and presenting new augmentation content based on the newmetrics. For example, if an augmented content describing certain publicpolicy information is promoted, then that augmented content's priorityis increased, and the priorities of all augmented content that referencesome public policy, or government policy get increased. In addition, theaugmented content can be dynamically updated based on user interaction,e.g. selection and/or clicking, within the reference or augmentedcontent in real time. There are various means to implement the augmentedcontent presentation layers such as dials for global and local augmentedcontent, or a scroll-area of small windows for various augmentedcontent. Describing all these various means to implement the augmentedcontent presentation layer is not necessary to understand thisdisclosure. Furthermore, a person skilled in the art would understandand would be able to employ many different means to implement augmentedcontent presentation layers without departing from the spirit of thisdisclosure.

In accordance with one embodiment, while generating augmented contentmay result in a lot of data that cannot be shown on the display, thisdata can be stored in a deep queue. A deep queue means that there ismore augmented content (data) in the queue than what is displayed on thescreen. For example, not all mined augmented content can be displayedsimultaneously due to physical screen size limitations or the displaylayer size. A user can hover over the queue or press an arrow to scrollthrough the augmented content in the queue. In addition, it is importantto note that the augmented content being presented to the user maycomprise actual data, snap shot of the actual data, a processed portionof the actual data, or a link to the location where the actual data canbe retrieved.

Theme-based augmented content can further enhance a user's experience bypresenting a set of themes. In accordance with one embodiment, when theuser selects or deselects a theme, a new or updated augmented content ispresented to the user. An option to expedite augmentation and improvethe quality is to rely on the user's preferences and feedback. When theapplication is invoked, a set of categories/themes can be presented tothe user. These constitute meta-data. By relying on the user choices ofthemes, augmentation can be enhanced and filtered. For example, aresearch paper that deals with AIDS virus would trigger a set of themessuch as Pharmaceuticals; Discrimination, etc . . . The user who isinterested in science and pharmacology but not in the social aspectsrelated to AIDS would deselect ‘Discrimination’. Thus, all augmentedcontent presented will be tailored to refer to categories that arerelated to science and other related aspects of the research. The themecan further be defined by a category or a set of related categories.This will serve to prune the augmented data and only present therelevant data that is of interest to the user and the task he iscarrying out at that moment.

Multilevel context sensitive augmented content application can beimplemented as a stand-alone application, on top of another application,or as an extension for applications, e.g. a browser extension. Inaccordance with one embodiment, further refinement or fine tuning ofvarious options for customization of augmentation system such asaggregating, mining, filtering, and presenting various aspect of data ormetadata can be performed dynamically in real-time. In addition, thecustomization of augmentation system maybe performed based on at leastone or more of a user's feedback, behavior, attributes, characteristics,theme, topics, and interests. Also when augmentation system presents alist of tags/categories, the user can provide feedback in the form orliking/disliking the tag. This is similar to promoting or dismissing anaugmented content. Therefore, in accordance with one embodiment, theaugmented content can be updated live. Furthermore, this user's feedbackwould also result in updating various subsystems such as the underlyingdata-mining, statistical computing algorithms, or machine-learningalgorithms or other information retrieval algorithms or heuristics.These updated subsystems are used to generate or create new signatures,metrics, or features which are based on user's feedback, e.g.liked/disliked tags, where the new signatures are used to generate newaugmented content or update the currently presented augmented content.

In accordance with one embodiment, a feature of a system for generatingand presenting multilevel context sensitive augmented content is theability to utilize online and offline mining and analytics foraugmentation. For example, mining and processing in real-time or inbatch mode and store data in a data store (local or remote) orpresenting real-time augmented content to the user. The stored data canbe used for future augmentation. Metadata and other relevant dataelements can also be annotated in real-time to capture user'spreferences and experiences. In addition, metadata and other relevantdata elements can be stored in a central repository to be leveraged forfuture augmentation of same or similar content. A brief description ofmetadata is that it is data that describes other data. For example:‘public health’ is a category that encompasses diseases. This higherlevel category ‘public health’ is a metadata for diseases.

In accordance with one embodiment, a multilevel context sensitiveaugmented content system uses at least two levels, a global level and alocal level. The following explains the difference between global andlocal augmented content. Global augmented content refers to augmenteddata that pertain to the overall document that the user is currentlybrowsing, exploring, or interacting with. A local augmented content canrefer to augmented content based on a particular piece, paragraph,sentence, word, image, icon, symbol, etc . . . of that document that theuser is currently browsing, exploring, or interacting with. Global &local augmented content are presented using a dynamic deep queue, andthe user can control the displaying of at least a portion of theaugmented content. Content sources for augmentation can be provided frommany sources. An example of such content sources includes but is notlimited to a user's own documents and data on desktop, web-content,social media sites, enterprise data-marts, and local and remote datastores, ontologies, other categorization, and/or semantic orrelationship graphs.

The multilevel context sensitive augmented content can be successfullyimplemented to augment a user's browsing experience as discussed above.In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content can besuccessfully implemented as an application for augmented user experience(UX). The system can increase productivity, provides augmenteddata-mining & data-exploration platform, augmented e-learning ande-research system, augmented desktop-based & mobile-based browsing,exploration, research, discovery, and learning platforms, dataaugmentation for better healthcare products and services, dataaugmentation for better educational products and services, augmentationsystem for better content management and relationship platform for bothenterprise and consumer applications, enhanced online-shopping researchand UX, enhanced marketing campaigns, an enhanced news access UX are butto name a few of application benefiting from a system for generating andpresenting multilevel context sensitive augmented content.

Semantic processing is the process of reasoning about the underlyingconcepts and expressing their relationships. In addition to variousaugmentation methods as described above, the following semantic basedtechniques can also be used in a system for generating and presentingaugmented content. In accordance with one embodiment, utilizing existingtags in public sources, utilizing batch-processed tags as a cloudapplication, semantic processing of selected content to generate a matchto an existing tag, semantic processing to generate augmented content onthe fly and utilizing user's feedback for promoting and dismissingaugmented content are but examples for methods to provide a betteruser-relevant augmented content. Generating augmented content on the flycan also be accomplished by using a feedback mechanism provided by theuser to enable mining and generating of new augmented data to bepresented to the user.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content is used toimprove the analytics of large data sets by leveraging pre-processeddata and already generated relationships. Given a content that is theresult of a statistical data mining and exploration phase on a small orlarge amounts data—be it remote or local—extract the correlation metricsand other signatures that demonstrate a meta-relationship and leverageit in other data-mining, analytics, and to generate augmented content.For example: When a user presents some key words to a search engine, theuser gets a set of links that are related in addition to some ads thatcould very well be related to the key words you have entered or to somepersonal data known or extracted of the user. These presented links andads have gone through a huge amount of processing and computation in thecloud. By knowing that a relationship or a meta-relationship existsbetween the keywords, links, and may be other content pushed to the userlike ads, the analytics operation can leverage them and extract, store,and leverage these signatures for future browsing or for presentingcontext sensitive augmented content.

In accordance with one embodiment, the content presented to the searchengine can be either parsed from the html or other format or interfaceproduced by a data provider. Or, it can be scanned through OCR if thedata format is encrypted. This ability to take a snap shot of a screenand analyzes and leverages its data and relationships empowers andsimplifies the augmentation and analytics processes and improves thethroughput since the signatures/correlation metrics extracted are aresult of processing a significantly smaller set of data. Therefore, theperformance gain of a system for generating and presenting multilevelcontext sensitive augmented content is orders of magnitude compared tomining massive data sets in the cloud.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content presents theaugmented content along with the reference content using two or moredifferent presentation layers displayed using the same display screen.In addition, the system provides the ability to customize the generationof augmented data in situ (in place) while working on original orreference content, where the augmented data can be displayed on see thrupresentation layers so as not to obscure the original or referencecontent and to maximize use of the display screen, and/or the displayingarea.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content utilizesdynamic updates of displayed augmented content using presentation layerswhile a user views and manipulates reference content displayed usinganother presentation layer. It is preferable to use a translucentpresentation layer for the augmented content presentation layer that islocated on top of the displayed reference content so that the user caneasily manipulate or interact with the reference content whilesimultaneously viewing the dynamically updated augmented content. As canbe easily appreciated by person skilled in the art that displayingrelevant augmented data in a separate tab or page would result in lossof context relationship and provides a less efficient and less friendlyuser experience. Similarly, displaying the augmented content on thesidebars is possible as well. However it consumes screen space andhinders displaying of the reference content. Therefore, the ability tokeep the reference content accessible to the user while displaying theaugmented data on top of the original content provides a much smootherand efficient user experience. Furthermore, the user can easily hide,size, move, or display the augmented content without affecting thereference content.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content enables a userthe ability to associate any of the augmented content with the referencecontent or an attribute of the reference content source using one ormore types of metadata. The system enables the user to save theassociated metadata for future use or sessions. For example, theassociation of metadata can be accomplished by embedding a link in thetext, by associating a link with a text, or by associating any data ormetadata with the reference content or any part of the referencecontent. Moreover, the user has the ability to specify a category ormore as a source or criterion of augmentation. The user can also defineassociation rules that join a group of attributes, categories, and othermetrics together to provide a richer input to aid the augmentationsystem to generate more relevant augmentation content. For example, anenterprise sales projection document can always be augmented with anydata source or data documents that generated the projection. Thecriterion is a category that says source sales data and not necessarilythe exact data documents. The sales data can be extracted automaticallyby the augmentation system. Utilizing selected or provided categories ofinterest, the augmentation system can carry out an updating procedurefor any associated data or metadata for any other reference content.Furthermore, the augmented content is displayed using see-through layersso that the user always sees and has access to the original or referencecontent. The user is able to access, browse, move, select, hide, tap,scroll, or interact with the reference or augmented content while thesystem dynamically generates and displays an updated augmented contentusing the augmentation presentation layer. It is noted that the userinteraction with the reference or augmented content can result in havinga new reference content that the user wishes to interact with, hence, anew augmented content is generated and displayed. The system keeps trackof and saves certain information regarding this nested augmentationlevel. The system provides the user the ability to switch back and forthbetween various nested augmentation levels as well as saving or sharingthe augmentation filters or settings used for a particular session.

In accordance with one embodiment, further enhancement of the userexperience is achieved by enabling the user to change the skin (or lookof a user interface UI) of the augmentation system. For example, thesame components of a UI (buttons, options, data) can be displayed on thescreen in a variety of ways. Usually, a library of templates and coloroptions can be provided to allow the user to customize the augmentedcontent presented by the application. In addition, the global augmentedcontent and local augmented content can be displayed using one or moredifferent regions of the screen, or displaying the global and locallinks to the augmented content in two concentric circles around thereference content. The enhancement of the user experience is achieved byenabling the user to choose the most efficient way for that user toutilize the augmented content.

In accordance with one embodiment, user selectable skins can also beused to cover or hide pushed content that may exist or embedded in thereference content being viewed. User selectable areas of a skin can beused to enable the display of user selected content such as images oraugmented content, or pushed content such as advertisement. For example,an ad for tickets to a local concert when the user is browsing aspecific artist, or an ad for a book that relates to a global or localaugmented content of the user reference or currently viewed augmentedcontent, or any other monetization mechanism based on the augmentationprocess. The enhancement of the user experience includes a nestedmultilevel context sensitive augmented content where the augmentedcontent presented to the user can be further enhanced as a function ofthe various nested levels. The augmented content is presented whilekeeping track of the current content being viewed in relationship to theoriginal content that the user started with and all levels in between.This provides a hierarchical augmentation system that enables the userto access and build nested levels of augmentation.

In accordance with one embodiment, the user interface, or UI, for asystem for generating and presenting multilevel context sensitiveaugmented content can be launched or started automatically and stayshidden from view until the user invoke a predefined programming functionto enable the UI functionality. For example, a single tap, hot-key,function-key, a gesture, or a multiple or a combination of actions actedupon a content would cause the transparent augmentation layer to beshown with the augmented content and in accordance with userpreferences, such as tags, skins, themes, etc . . . Selecting contentpresents or updates the augmented content already presented. Visiting anaugmentation link results in completely or partially (split screen)covering the reference content or original layer comprising the originalcontent. The UI provides the user the ability to navigate nestedaugmented content or jump back to reference or original content.

In accordance with one embodiment, additional UI features can further beused to increase the overall efficiency and provide a better userexperience. For example, saving the augmented content metrics in userhistory, and using history to enhance and/or tailor analytics andaugmentation as would be more relevant to each individual user or groupof users such as in corporate environment. Metrics here refer to thegenerated signatures as mentioned above. Also, it refers to anyannotations that are provided by the user such as priority,liking/promoting an augmented content or dismissing it. This can bestored for future sessions as well as using the augmented contentpromotion and dismissal to enhance augmentation in real time. Usingskins that cover an undesirable part of the screen, e.g. side columnswhere ads are pushed. The skin may be used for further customization ofthe viewed screen and potentially could be monetized and leveraged topresent relevant augmented content that is paid for by the user, such asads for objects, e.g. books, related to the content of a referencearticle.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content providesdynamic user-guided and customized context-sensitive data augmentationto facilitate learning, exploration and knowledge discovery. The systemprovides simultaneous interaction with the augmentation layer and thecontent layer. The system generates augmentation data based onuser-defined metrics and filters such as themes, categories of interest,document content and/or part of it. The generated data is not a rigidaugmented content. The generated augmented content is any data, concept,and relationships that are presented as a result of the data mining andprocessing of the original content and the user-defined metrics andfilters.

In accordance with one embodiment, the system utilizes dynamic andinteractive methods to successively refine and tailor the augmentedcontent based on a user's guidelines, filters, and metrics. The systemrelies on a variety of sources for content augmentation by accessing anyonline or offline databases, crowd-sourced databases, or open databases.Furthermore, over time, a custom built graph of concepts andrelationships can be built between different pieces of data as they areprocessed and augmented based on the user's filters and metrics toimprove the performance of the system and the User Experience. Thesystem provides a context-sensitive hierarchical augmentation frameworkfor deeper and expansive exploration and knowledge discovery. The systemenables construction of a customized graph of data, concepts, andrelationships based on the filters and metrics provided even in theabsence of content. Content can be generated on the fly for furtherexploration.

In accordance with one embodiment, the system enables sharing ofaugmented data and the associated metrics that generated them. Thisenables richer knowledge discovery by further refining a user'saugmented data based on other users' augmented content. This is usefulfor collaborative research and knowledge discovery. The system can belaunched from offline and online documents or reference content togenerate the augmentation content, data and graph of relationshipsamongst the concepts represented by the augmented content.

In accordance with one embodiment, the system provides a UI to displayand manipulate reference content and augmented content concurrently,dynamically, and interactively. The system provides one ore moretranslucent layers on top of the reference content to show the augmentedcontent. Translucent layers facilitate displaying the reference contentas well as the augmented content. Translucent layers can fully orpartially cover the original content. Augmentation layers can be hidden,minimized ((shown as icon), or moved around on the display screen tofacilitate easier display and interaction with the reference content.The system enables the user to manipulate and control a set of displaylayers (reference content layer, and/or augmentation display layers) ina very flexible fashion such that the user can size up, down, move,show, hide any of those display layers. The system provides anintuitive, rich, and friendly UX for data exploration and knowledgediscovery on small and large display screens. In particular, displayingof the augmented content concurrently and interactively on the originalcontent empowers the user to use this system on smart phones, tablets,and any other display. Furthermore, the system provides means to insertadditional content on the augmentation layers based on analytics on theaugmented content and the original content.

Knowledge discovery system serves to augment, clarify, enrich, andexpand on a relevant topic or topics in a document. A number ofinformation processing techniques is carried out to disambiguateinformation and extract names, concepts, events, and other relevantmeta-data using Name-Entity-Recognition (NER), topic modeling todiscover topics related to the reference document. Such topics can beeither explicitly mentioned or discovered by relying on techniques basedon information processing, data mining, machine learning to processdiscovery patterns, and causality graph and other web and datarepositories.

Name Entity Resolution/Recognition (NER) processes the document,disambiguates names and concepts, extracts names, concepts, dates, namephrases, and any other data that can be parsed, processed, or inferred.Use of latest information extraction, data mining and natural processingare some of the techniques and algorithms that can be used in this step.

Topic Modeling and Topic Graph: Mine data from stored knowledge graphs,causality graphs, or other repositories to extract and cluster topicsand categories from the mined names, concepts, and other processed data.Latest research in data clustering, topic extraction, inference,modeling, and latent topic discovery can be used to build a topic graph.A topic and clusters are interchangeable in this graph. A cluster is aset of related data that share a set of common features andrelationships. One of those features or relationships can be a theme. Atopic is a cluster of documents that share a common theme. Not allrelevant topics can be discovered by the topic extraction step. Moretopics (relevant and possible hidden) can be extracted by the aid of thediscovery patterns and causality graphs below.

Hierarchical Graph Discovery of intermediate topics and themes todiscover/expose relationship between related topics and clusters. Thisgraph can be a pre-defined taxonomy, or a hierarchically constructedgraph based on different levels of coarse and fine clusters constructedbased on the available content.

Data Clustering is the process of constructing a set of clusters ofrelated documents. The relatedness is defined based on a set of desiredfeatures and/or relationships. Topic Modeling above is a form of dataclusters where each topic is a cluster that shares a common theme.

Discovery Patterns (DP) 100 are templates that aid the discovery systemto extract the relevant knowledge for a topic or a concept, as shown inFIG. 1. For example, for a topic, it will query for the relevantproperties or relationships that are annotated on the topic or itsmeta-topic. Furthermore, a DP can help in defining a set of competencyquestions that will be very important to data augmentation and knowledgediscovery. DPs define and a set of Competency Questions (CQ) that can beextracted from the data/content to extract and discover salient contentand relationships. These discovery patterns can be pre-defined, manuallyconstructed, extracted from other data sites or repositories, or craftedon the fly. They can also be further enhanced and massaged as the systemgathers more data. Also, they can be tailored based on user's specificfeatures and interests.

Competency Questions (CQ) define a set of queries that are very specificto the content at hand. These queries enable very focused knowledgediscovery. These CQs are domain dependent. For example the set of CQsfor knowledge discovery of legal corpus is different from the set of CQsfor knowledge discovery of medical corpus. Our system in addition toleveraging pre-defined CQs modeled in pre-defined DPs, it enables theuser to provide a custom-defined DPs their associated CQs, and alsoautomatically infer a set of CQs and dynamically construct a set of DPsbased on the available content and features.

Ontologies and public repositories provide pre-defined sets of conceptsand relationships that can be leveraged in the knowledge discoveryprocess. Wikipedia, Wordnet, Freebase, Verbnet are examples of suchrepositories that are rich, and constantly updated. Although theseontologies and repositories are bulky, they are rich with relevantcontent. Our system leverages these repositories amongst others sourcesto discover rich augmentation content.

Causality graph (CG) enriches and enhances the knowledge discoveryphase. By minding known data (world wide web documents), accessiblerepositories (public and possible private), a large body of knowledgecan be modeled. CG serves to build the set of relationship that existbetween the topics can be extracted and modeled in the CG. Also,causality and dates of events can be extracted and inferred and modeledin the CG. These will serve in discovering more hidden but importanttopics and relationships that should exist in the topic graph but theyhave not been discovered yet.

Abstracted Causality Graph is a graph that is constructed from thecausality graph (CG), the CG should be abstracted so that similartopics, relationships, cause and effects, and their meta-subjects arecaptured. This will aid in leveraging all this knowledge to augment andenrich new information and knowledge. This is essential for knowledgediscovery. An example of abstracted concepts could be if company Xacquires company Y, it is not important who X and Y are, but what isvery important is the notion that a company can acquire another company.This way when we see a name of company in a new document, we canautomatically ask the question about any prior or expected acquisitionfor the company at hand.

Data Augmentation as was discussed in a previous disclosure refers tolocal and global data augmentation and knowledge that are extracted andpresented to the user to further expand on the document at hand. Thisdata is based on all the knowledge modeled in the discovery graph(topics, clusters, and relationships), causality graph, discoverypatterns library, and other on the fly information extraction. Thisaugmentation will facilitate Timeline Events related to both local andglobal augmentation data and will provide a rich knowledge discoveryexperience. The user can browse in time to discover relevant knowledgeabout the topic or topics at hand.

A block diagram for an Augmentation System 1500 is shown in FIG. 2. Thegoal of this system is to read, synthesize, and/or extract a set ofcompetency questions that will enable smarter content discovery andaugmentation. Competency Questions (CQ) is a set of queries that arevery specific, well defined, and rich features that guide the knowledgediscovery process. Box 1510 reads and processes a set of ‘competencyquestions’ from the user, Box 1520 synthesizes those CQs based on anautomated template generation that will define and fill the relevantcompetency questions for the content at hand. Synthesis based onextracted relationships of topics in CG, topical graphs, or othersynthesized relationships. Box 1530 extracts those CQ based on contentdiscovered by processing those relevant topics and entities. Box 1540compiles and outputs the constructed set of CQs as produced by any orall boxes 1510, 1520, and 1530.

A block diagram for Causality and Augmentation System 1600 is shown inFIG. 3. The goal of this system is to construct a causality graph thatcaptures the cause-effect between different mined or discovered topicsin the system. This causality relationship extraction adds anotherdimension to the knowledge discovery system. Box 1610 defines a set ofprior topics that are relevant to content. Box 1620 defines a referencetopic to be augmented. Box 1630 defines a set of topics that are causedby the prior topics and related to the current reference topic. Box 1640defines the set of actors that are at play in the causal relationships.These actors can be named entities such (person, location,organizations, groups). Box 1650 defines a set of topics and theirrelated categories so that the correct causal relationship is usedshould there be more than one relationship. Box 1660 a set of DiscoveryPatterns (DP) that will enable the system to extract the right meta dataand annotations when discovering the causal relationships. Box 1680defines a set of user-provided features that will aid in this discoveryprocess. Box 1670 defines that causality graph that is the result ofprocessing all the input defined in the previous mentioned boxes. Box1690 is the set of causal relationships and relevant content to be addedto the augmented content.

A block diagram of a Causality Graph Synthesis 1700 is shown in FIG. 4.The goal of this system is to build and abstract the causality graphsuch that it is applicable to a different set of actors and entitiesthat share the same set of relationships defined in the graph. Box 1710defines a set of topics and relevant content that can be mined form anysource public or private. These constitute the nodes in the causalitygraph. This system builds and infers edges between those nodes based ona set of rules, heuristics, discovered relationships, or pre-definedrelationships. Box 1720 extracts relationships between the presentedentities. Box 1730 extracts relationships between the presented topicsand their corresponding categories, and Box 1740 extracts instances ofcausalities based on the presented content itself. Box 1760 checks theexisting causality graph for the discovered or inferred edges. If theyare not present already, they are added to the causality graph (CG). Box1750 processes the updated causality graph (CG) and infers abstractedrelationships and adds it to the graph so that the CG becomes moreabstract and applicable to future instances of relevant topics andentities. Box 1770 is the output of this system that presents a rich andabstract causality graph.

A block diagram of the Augmentation System with the causality graph 1800is shown in FIG. 5. This block diagram shows an overview of the wholeaugmentation system operation. Block 900 shows the document that needsto be augmented. Box 905 shows the features that were extracted fromthis document as signatures to aid in finding relevant augmentation. Box910 defines the named-entity-recognition system what extracts thesalient entities in the system. Box 915 presents the set of entitiesextracted. Box 920 presents other features or properties such as datesor others that will further aid in augmentation. Box 930 shows a rankingengine for the features presented so that noisy or less salient featuresare pruned out to further aid in higher quality augmentation content.Clustering and Topic modeling is executed on this relevant content inBox 935. Box 940 presents the set of relevant clusters and topics thatare constructed. Leveraging a library of pre-defined (Box 960),dynamically synthesized (Box 950, Box 955), or user-provided (Box 965)discovery patterns carry out further content augmentation. Box 955defines a mapping between a discovery pattern in the library and asynthesized relationship based on the presented features. Box 970presents the resultant set of discovery patterns. Box 975 processesthose patterns by examining the causality graph (CG) to see if suchrelationships exist or are defined. Further augmentation content can beadded to the causality graph by processing relevant documents in publicor private repositories (Box 985). Box 990 presents a new set ofentities and topics from the freshly mined content. Box 995 extracts atimeline from the freshly mined content so that the right part of thecausality graph is updated. This data is further used to extract arelevant timeline to process the dates and timeline to link the relevanttopics together (Box 995). The data in Boxes 990 and 995 are furtherutilized to infer and extract more knowledge from the Causality Graph inBox 100. Box 100 presents the new augmentation content that will beadded to the causality graph. At the end of this process, a rich set oflocal and global augmentation along with a knowledge graph with atimeline that connect the different topics (local and global) and themined relationships and properties will be available.

A block diagram of an Augmentation System 100 is shown in FIG. 6. AReference Content 105 corresponds to any electronic document or web pagethat a user wants to invoke the Augmentation System 100 to get AugmentedContent 190. The Reference Content 105 can be stored locally in a memorysubsystem of an electronic device, a memory subsystem of a displayscreen device, or is accessed from a remote location via a wired orwireless communication system. The communication system could use theinternet, a cloud, a data store, a computing device, server or adatabase via a wired or wireless networking link. The augmented contentis a generated content by the Augmentation System 100 based on theReference Content 105 using a set of features, filters, and categorieswhich are produced by at least one of an Extract Features 120, ExtractCategories 125, and Update Categories 137 subsystems as shown in FIG. 6.

A Local Content 110 is a selected portion of the Reference Content 105which the user wishes to get more specific augmentation about, or thatis a portion of the Reference Content 105 that the user is interactingwith. Furthermore, the Local Content 110 may also be automaticallyselected, tagged, managed, or generated by the Augmentation System 100,e.g. based on a displayed portion of the Reference Content 105 or a userinteraction with a portion of the Reference Content 105. Furthermore,the presentation and/or the displaying of the Augmented Content 190 ismanaged using Manage RAC 145 (RAC refers to Relevant Augmented Content)to control a Display Queue 165 and Display RAC 170.

The Augmentation System 100 generates Augmented Content 190 byfacilitating the construction of a user-customized network of concepts,objects and relationships that serve to augment the Reference Content105 at hand for the purpose of knowledge discovery, learning, and aricher user experience in browsing and/or interacting with datainformation. This Augmentation System 100 generates any one of a networkof concepts, a network of objects, and a network of relationships usingone or more of a set of features, a set of filters, and a set ofcategories. Each of the set of features, the set of filters, and the setof categories can be customized and tailored based on the user'sinterests and input. The constructed network can be saved and furtheraugmented over time for richer and more efficient user experience.

The Extract Features 120 subsystem extracts a set of features from theReference Content 105. General Features 117 can provide a set offeatures that can be updated and tailored overtime to at least one of aspecific user, specific project, specific objective, and specificsubject. Extract Features 120 generates a set of filters that denotesthe desired concepts for augmentation. For example, these concepts couldbe names of people, history, events, topics, or other meta-data. Thesedata are either computed on the fly or pre-computed and stored locallyor remotely for current or subsequent augmentation sessions. Thisextraction process is based on embedded data in at least one of theReference Content 105, in a linked content to the Reference Content 105,metadata of the Reference Content 105, e.g. a title of the ReferenceContent 105, linked content to the Local Content 110, and semanticinformation that are either associated with the Reference Content 105 orthat can be extracted/aggregated from the Reference Content 105. Otherdata that can be extracted or inferred can be further used forconstructing a more meaningful feature set by utilizing a variety ofinformation retrieval, extraction, and inference algorithms and methods.There is large body of work on feature extraction that utilizes thecloud as well as other large-scale solutions. These approaches can beleveraged by the Extract Features 120 along with flexible and efficientalgorithms to generate a feature set on the fly based on the metrics andsignatures mentioned earlier. Furthermore, any part of the AugmentationSystem 100 can be run remotely on a server or in the clouds, or it canbe run locally on the host device.

The Extract Categories 125 function uses a set of categories or topicsthat are extracted based on the data that can be associated or extractedfrom the Reference Content 105. This data can be either meta-data or anyother related data to the Reference Content 105. The Extract Categories125 extracts a set of categories from the Reference Content 105 and itsassociated links and data. Also, the system utilizes any embeddedcategories or meta-data that are either embedded in the link or attachedto the Reference Content 105. The extracted categories can also describemeta-data about the topic at hand. For example, if the reference contentis an article about AIDS, there are many categories that can augmentdata about AIDS. For example, a set of categories can be: History ofAIDS, Science of AIDS, Social Impact of AIDS, Symptoms of AIDS, etc . .. A user may only be interested in the science of AIDS, so a user willinteract with the presented categories, e.g. by deselecting allcategories that are not related to science, and this will impact the setof features that are used in augmenting the Reference Content 105. Otherdata that can be extracted or inferred can be further used forconstructing a more meaningful category set by utilizing a variety ofinformation retrieval, extraction, and inference algorithms and methods.In addition, a General Categories 115, as shown in FIG. 6, is a set ofdefault categories that the Update Categories 137 processes to reflectthe user's interests. For example, the General Categories 115 can beBusiness, Politics, Education, Research, Health, Technology, etc . . .The Update Categories 137 may use this optional input from the user tobias the augmentation to the categories of interest. This optional inputcan be stored and updated over time.

The interaction of a user with the Augmented Content 190 maybeaccomplished in a variety of ways. For example, the user may select oneor more of the presented categories for removal, selection, decreasingpriority, and increasing priority. The user may also define, modify, orinteract with an association rule to aid Extract Features 120 togenerate a more useful set of filters for better augmented content. Theassociation rule can leverage, use, or joins one or more categories,features, filters, or concepts to (i) generate a new set of features,filters, categories, or Augmented Content 190, and (ii) to modify one ormore of the set of features, filters, or categories which are being usedto generate the Augmented Content 190. Based on the General Categories115 and user's interaction, further categorization and featureextraction will be biased towards the user's interaction or input. Thisis an optional input that is used to customize the Augmented Content 190based on a user's needs, the user's interaction with Augmented Content190, or to aid the Augmentation System 100 to provide more relevantAugmented Content 190 for a specific purpose. Upon a user's interactionwith the Augmented Content 190, an Update 130 function enables theuser's input to be considered by Update Categories 137, e.g. a user maychoose to delete some of the default/general categories that are not ofinterest or to elevate the priorities of some of those categories. Whendeleting categories, the Update Categories 137 will reduce the weight ofthe features that are related to those categories. When categories areelevated in priority, the Update Categories 137 increases the weightgiven to those features that are related to those categories. Thus,affecting and updating the Augmented Content 190 presented to the user.

An Update Filters 150 is used to indicate a user's preference for afeature or automatic feedback based on user's interaction with theAugmented Content 190. For example, when one or more of the ReferenceContent 105, Local Content 110, and Augmented Content 190 get updated orinteracted with by a user, then more clues and feedback can be gatheredfrom the updated list or the user's interaction as to revise thefeatures and categories that are of interest to the user in real time.However, the user may choose not to update the features and categories,and the Augmentation System 100 provides the user the ability to controlhow and when the Augmented Content 190 is generated and/or updated.

An Update Features & Categories 135 subsystem receives a first set offeatures from the Extract Features 120 subsystem, a first set ofcategories from the Extract Categories 125, and/or an updated set ofcategories from the Update Categories 137, and/or an Update Filters 150.Update Features & Categories 135 manages and controls the updating ofthe actual features and categories sets including any decision makingbased on the user input or interaction. The Update Features & Categories135 may communicate with any one of Extract Features 120, ExtractCategories 125, and Update Categories 137 to generate more features andcategories based on a variety of parameters including the user'spreferences. Furthermore, Update Features & Categories 135 also handlesupdating relationships and cleaning up for those features and categoriesthat were updated by the user.

A Compile RAC 140 subsystem receives a set categories and a set featuresfrom the Update Features & Categories 135 subsystem. Compile RAC 140includes a variety of functions and algorithms such as machine-learning,data mining and extraction, web crawling, data-mart accessing,extraction and processing functions, and other intelligent algorithmsand approaches are used to compile a set of relevant augmented contentor pages (RACs) based on at least one of the Reference Content 105,Local Content 110, and the interest of the user. Managed RAC 145subsystem is the controller that manages the presentation of theAugmented Content 190 via a Display Queue 165 and Display RAC 170. TheAugmentation System 100 listens to inputs from the user and manages thegeneration of the Augmented Content 190. The Managed RAC 145 subsystemgenerates three outputs taking into consideration a user's feedback orinput. The Managed RAC 145 subsystem generates and controls thecommunication of the generated Augmented Content 190 using Display Queue165 and Display RAC 170. In addition, Managed RAC 145 generates anupdate request to Update RAC 155 for any necessary update to the DisplayQueue 165 based on a user's interaction or input. The Display Queue 165displays in a desired skin at least a portion of the queue of RACs sothat the user can browse through them and select some to view. TheDisplay Queue 165 displays a link, a summary, or a portion of thecompiled relevant content or pages. Upon selection or interaction by auser with one of the displayed RACs, the Display RAC 170 retrieves therespective relevant page RAC and displays at least a portion of it. TheDisplay RAC 170 subsystem manages and controls the displaying of theAugmented Content 190 using the display screen. Display RAC 170 can useone or more display layers on top of the Reference Content 105 or LocalContent 110 via translucent display layers as discussed in previousparagraphs.

A block diagram of a Hierarchical Augmentation System 200 is shown inFIG. 7. This hierarchical augmentation or nested augmentation capabilityenables a user to augment any content that is the result of dataaugmentation at any level of browsing or exploration. For example, giventhat the Augmentation System 210 generates a list of RACs, the user mayselect any one of the RACs or a group of RACs to invoke the augmentationsystem on and to generate another level of augmentation. TheAugmentation System 200 allows the user to go back and forth in thehierarchical graph to browse any particular content at any level, be ita reference or augmented content. For example, the Augmentation System200 provides augmented content at Process 1 220, which is the firstinvocation of the augmentation system on reference content, a user mayelect to augment one or more of the augmented content of Process 1 220.The Augmentation System 200 uses the elected content to be augmentedfrom Process 1 220 as an input or reference content to Process 2 230 foraugmentation. Process 2 230, which is considered the second invocationof the augmentation system on a reference content, generates in turnaugmented content which the user can further refine or interact with,and so on for Process K 340, Process (n−1) 350, and Process (n) 360.Multilevel nesting or hierarchical augmentation is not limited to aspecific number of levels. Of course certain hardware or softwarelimitations or a particular application may dictate the use of aspecific number of levels. However, this is an option that can be usedto various extents as part of the customization of Augmentation System200 for any particular usage.

A block diagram of a Hierarchical Augmentation System 300 is shown inFIG. 8. This hierarchical augmentation or nested augmentation capabilitycomprises the same capabilities as the Hierarchical Augmentation System200 is shown in FIG. 7 and includes an Augmentation System Control 390subsystem that is communicating Augmented and Reference contents AR-325,AR-335, AR-345, AR355, and AR-365 with Process 1 320, Process 2 330,Process K 340, Process (n−1) 350, and Process (n) 360, respectively. Asdescribed above Process 1 320 corresponds to a first level instance ofAugmented System 100, and Process (n) corresponds to an n-th levelinstance of Augmented System 100. Given that each of the nestedaugmented systems may generate different augmentation content for eachhierarchical level at least due to variations in user input or thereference content corresponding to the hierarchical level, theAugmentation System Control 390 may receive one or more of the generatedaugmented content of each hierarchical level, a copy of the set offilters, a copy of the set of features, and a copy of the set ofcategories. The Augmentation System Control 390 can further runsophisticated statistics, analytics and algorithms to extract newfeatures or generate new filters or categories. Furthermore, theAugmentation System Control 390 may receive user input to control whattype of analysis or augmentation the user expects the HierarchicalAugmentation System 300 to provide or keep track of nested contents thatthe user is interacting with, viewing, or manipulating at various levelsof hierarchy.

A block diagram of an Augmentation System 400 using a Display Control420 subsystem is shown in FIG. 9. The Augmentation System 410 isessentially the same as any one of the Augmentation System 100,Augmentation System 200 and Augmentation System 300 as shown in FIG. 6,FIG. 7, and FIG. 8 respectively. The Display Control 420 subsystemcontrols the displaying of various elements such as Augmented Content450 and Reference Content 440, which are output of the AugmentationSystem 410. In addition, the Display Control 420 receives input controlfrom Augmentation Display 430 subsystem and/or from a user interactingwith the Augmentation Display 430 or one or more display layersdisplayed using the Augmentation Display 430. Based on the AugmentedContent 450 generated from Augmentation System 410, Display Control 420generates and/or controls different display layers, widgets, icons, andother knobs which are utilized to show, control, or manipulate any oneof the Augmented Content 450 and Reference Content 440. Furthermore,Display Control 420 provides means for the user to interact with any oneof the Reference Content 440 or the Augmented Content 450.

In accordance with one embodiment, the Augmentation System 410, theDisplay Control 420, and Augmentation Display 430 are elements of thesame physical electronic system such as a mobile device. The user canmanipulate any one of the Augmented Content 450, Reference Content 440,and how each is displayed onto the Augmentation Display 430. Inaddition, a user interface (UI) may be used to further aid the user tomanipulate or interact with any one of the Reference Content 440 and theAugmented Content 450 and the displaying of such content. Furthermore,the UI can provide an easy mechanism for a user to interact with thecategories, widgets, buttons, and any other option that is presented forthe user to engage with the Augmentation System 410.

In accordance with one embodiment, the Augmentation System 410, and theDisplay Control 420 are elements of a first electronic device that isseparate from a second electronic device comprising the AugmentationDisplay 430, wherein the first and second electronic devices communicatethe Reference Content 440 and the Augmented Content 450 back and forthbased on the Augmentation System 410 and/or a user interaction with anyone of Reference Content 440 and Augmented Content 450.

In accordance with one embodiment, the Augmentation Display 430, and theDisplay Control 420 are elements of a first electronic device that isseparate from a second electronic device comprising the AugmentationSystem 410, wherein the first and second electronic devices communicatethe Reference Content 440 and the Augmented Content 450 back and forthbased on the Augmentation System 410 and/or a user interaction with anyone of Reference Content 440 and Augmented Content 450.

In accordance with one embodiment, a system for extraction andgeneration of features and categories Extract Relevant Features 500 isshown in FIG. 10. The Extract Relevant Features 500 is tasked withbuilding a set of features and categories that any one of theAugmentation System 100, Augmentation System 200, Augmentation System300 and Augmentation System 400) can utilize to generate augmentedcontent. Reference Content 510 is similar to Reference Content 105, andLocal Content 520 is similar to Local Content 110. Categories 530 is asubsystem which is responsible for constructing a list of categoriesthat captures or is responsive to the user's inputs and preferences, aset of extracted categories from Reference Content 510 and Local Content520, and a set of customized categories associated with the user.Features and Metrics 525 is a subsystem which generates a set offeatures, a set of signatures, and/or a set of metrics each of which iseither dynamically generated or pre-computed and stored. Features andMetrics 525 delivers these sets of features to an Extract Features 540subsystem. In addition, the Extract Features 540 receives input fromReference Content 510, Local Content 520, Features and Metrics 525, andCategories 530. Extract Features 540 delivers a set of features, a setof signatures, and a set of metrics to Compile RACs 550 subsystem, whichin turn utilizes one or more of those sets to compile from the internet,a local data store, or any other data repository (public or private) aset of data elements. A Relevant Augmented Content 560 subsystemreceives the set of data elements and/or the set of features, the set ofsignatures, and the set of metrics to generate a customized augmentedcontent for the user.

In accordance with one embodiment, a simplified block diagram of asystem for extraction and generation of features and categories ExtractRelevant Features 600 is shown in FIG. 11. The Extract Relevant Features600 can be used as a part of an augmentation system such as AugmentationSystem 100, Augmentation System 200, Augmentation System 300 andAugmentation System 400 each of which has been described above. TheExtract Relevant Features 600 is utilized to compile a set of features,using Compile RAC 650, to be used by an augmentation system to generateaugmented content. Extract Candidate 618 processes at least a portion ofa Reference Content 608, and receives other user-provided input toextract or generate one or more set of filters and features. Features620 uses the one or more set of filters and features to organize, build,compile or store a user-customized network of features, concepts,objects and their relationships. Features 620 serve to provide a betterextraction or a focused extraction of a user's relevant set of featuresthat can provide a faster convergence on what the user is interested tosee or would want to see regarding the Reference Content 608. Inaddition, this provides a better value add augmented content for thepurpose of knowledge discovery, learning, and a richer user experiencein browsing and/or interacting with data information. Furthermore,Features 620 can learn, save and further refine the user-customizednetwork of features, concepts, objects and their relationships over timefor richer and more efficient user experience. Similarly, Categories 630uses the one or more set of filters and features generated by ExtractCandidate 618 to organize, build, compile or store a user-customizednetwork of categories and their relationships. Categories 630 can learn,save and further refine the user-customized network of categories andtheir relationships over time for richer and more efficient userexperience.

Metrics 640 is a system that can provide user influenced metricsinformation to Compile RAC 650. Metrics 640 uses the one or more set offilters and features generated by Extract Candidate 618 to organize,build, compile or store a user-customized network of metrics which canbe user defined or system's default. For example, Metrics 640 can usedate or time as a metric that can be used to further narrow and focus onthe relevance of the augmented content to the user or to the ReferenceContent 608. Another example is to use a source or a group of sources toaid Compile RAC 650 to limit or expand its compilation and generation ofrelevant augmented content. Metrics 640 can learn, save and furtherrefine the user-customized network of metrics and their relationshipsover time for richer and more efficient user experience. Metrics 640 canreceive real time information from the user or other part of anaugmentation system, and provides an update in real time to Compile RAC650.

Compile RAC 650 is used to compile the networks of features, categoriesand metrics received from Features 620, Categories 630, and Metrics 640to generate and prioritize a focused set of relevant augmented content(RAC) that captures the properties and/or attributes of ReferenceContent 608 and reflects the user's rules, interests, preferences, andattributes. This focused set of relevant augmented content (RAC) is tobe used by an augmentation system to deliver or present a concise andhighly relevant augmented content to the user. Compile RAC 650 is usedto resolve any conflicts that may exist between any of the networks offeatures, categories and metrics. Compile RAC 650 also provides anddetermines the priority of the final list of RACs to be delivered orpresented to the user. Compile RAC 650 can also receive, generate ormodify an association rule which can be used to leverage, or join one ormore categories, features, filters, concepts, or metrics to (i) generatea new set of features, filters, categories, or relevant augmentedcontent, and (ii) modify one or more of the set of features, filters, orcategories which are being used to generate the relevant augmentedcontent.

In accordance with one embodiment, an augmentation system can useDisplay Layers and Controls 700 as shown in FIG. 12 to display thegenerated Augmented Content 760, Global Augmented Content Queue 720,Global Augmented Content Queue 770, and the Reference Content 750. Forexample, this can be one instantiation of the data presentationmechanism of an augmentation system as described above, e.g AugmentationSystem 100. The user can change the look and feel (skin) of the DisplayLayers and Controls 700 using any number of skins (look and feeloptions). The Global Augmented Content Queue 720 corresponds to adisplayed part of a relevant augmented content (RAC) generated by theaugmentation system. The user can browse and scroll through this queueto select a relevant augmented content of interest. Local AugmentedContent Queue 770 refers to the relevant augmentation results that arerelated to the part of Reference Content 750 that the user hasinteracted with or is being displayed via Display Screen 710, and whichis referred to as Local Content. Display Layers and Controls 700 canmanage the display of the Global Augmented Content Queue 720 and LocalAugmented Content Queue 770 in various ways, such as the location of thedisplay of the queues as well as the portion of any one of the queuesthat is being displayed using Display Screen 710. For example, the usercan choose that only the Global Augmented Content Queue 720 isdisplayed, thus Display Layers and Controls 700 will manage to displaythe portion of RACs of the Global Augmented Content Queue 720 that maybeaccommodated onto the Display Screen 710. Similarly, the user may chooseto emphasize the Local Augmented Content Queue 720 and thus the DisplayLayers and Controls 700 will manage that as well. The Reference Content750 refers to the content being browsed and explored for furtheraugmentation. Display Screen 710 corresponds to a display screen thatmay be physically collocated within the same device where theaugmentation system is being used, or it can be part of a separateelectronic device. Augmented Content 760 is displayed using one or moredisplay layers, and is the augmentation content that the user chooses toview. An icon Promote 740 is used to highlight, select, or promote aspecific RAC. Promote 740 provides a mechanism for the user to interactwith any of the RACs of Global Augmented Content Queue 720 and LocalAugmented Content 770 by elevating the priorities of a RACs. Similarly,a demote icon (not shown) can be used by the user to remove or dismiss aRAC or a group of RACs entirely if the user is not interested in them.

A Simulated Display 800 is a use case scenario of the Display Layers andControls 700 and any one of the augmentation systems described earlieras shown in FIG. 13. This Simulated Display 800 presents an example of auser reading an article about AIDS as shown in Reference Content 810.The user then invokes the Augmentation System to augment ReferenceContent 810. Based on the categories presented to the user, the userselects categories that are related to AIDS Research and Science.Categories Related to AIDS RACs 820 shows part of the globalaugmentation deep queue that the system generated in response to theuser's interest in AIDS, Science, and Research.

A Simulated Display 900 is a use case scenario of the Display Layers andControls 700 and any one of the augmentation systems described earlieras shown in FIG. 14. This Simulated Display 900 presents an example of auser reading an article about AIDS as shown in Reference Content 910.Selected Content 920 shows an example of selecting part of the ReferenceContent 910. The user then invokes the Augmentation System to augmentReference Content 910 and Selected Content 920. Based on the user'schoice of categories related to Science and Research of AIDS, and thesystem's extracted categories and features RACs 905 shows part of theglobal augmentation deep queue that the system generated. Africa/IndiaAIDS RACs 930 shows part of the local augmentation deep queue that theAugmentation System generates in response to the user's selection ofpart of Reference Content 910. Augmented Display Layer 940 is an exampleof displaying of Africa/India AIDS RACs 930. Augmented Display Layer 940shows a RAC (HIV and AIDS) in the local augmentation queue that theelected to view.

New Work on Augmentation and Knowledge Discovery. Build on previous workof data augmentation and knowledge discovery. Tap into existingontologies. Utilized known, constructed, and synthesized DiscoveryPatterns to improve the performance of the system for online discoveryand mining. General Ontologies are bulky and inefficient to mine. It isbest the system does that only if Discovery Patterns are not availableto reduce the search space.

Mission: Provide an on-demand friendly and rich mobile knowledgediscovery platform that makes context-sensitive remote and hiddenrelevant knowledge and information accessible and useful. We willseamlessly and intuitively bring knowledge to every one.

Knowledge Discovery: The Next Revolution. Ongoing Progress to help usersfind information relevant to their immediate goals by improving searchand document classification. A huge gap between the way most systemsorganize information and the way humans wish to access that information.Search views information as sequences of words or numbers with no deepinterrelationships, while humans meaning conveyed by words. Humansexplore ideas and concepts, while automated systems are limited tosearching for words.

Knowledge Discovery Framework; Knowledge Discovery in Web Content;Relevant Topics Mining; Relevant Latent Topics Discovered; RelevantQueries Mining; Hierarchical Topics Graph, Discovery of intermediatetopics to discover/expose relationship between related topics; CausalityGraph Construction and Mining. New figures are added to utilizeDiscovery Patterns, Synthesize Causality Graph, and mining of CausalityGraph and other available repositories for a richer knowledge discoveryexperience. Not everything is available in causality graph and furtheronline mining for more augmentation data might be needed.

Name Entity Relationship: Named Entity Recognition (NER) is a key foraccurate content extraction for knowledge discovery. Personal names,places, dates, organizations, groups, parties and other named entities(NEs) to characterize topics in a document; Name Disambiguation; Namephrase parsing, compound names, . . . ; Known concepts, events, . . . .

Possible Product & Service Offerings: Knowledge Discovery platform thataids in any product or service where data augmentation and knowledgediscovery are desired or suitable. Examples: News discovery, Political,Business, Historical, Science, . . . ; Browsing & research; FinancialData Discovery; Company profile, competitive assessment, etc . . . ;eHealth Discovery; By leveraging a health-related DP; MedicationInformation, Patient Case Analysis, Prognosis and other related Data canbe discovered and displayed.

Discovery Patterns (DP): Pre-defined DP; Custom-tailored DP; On the flysynthesis of DP; Mapping info extracted from document/page to DP: MiningWeb for CQ (Competitive Questions); Extracting Relevant/latent Topics;Discover hidden and or nonobvious topics & relationships; Filling DPbased on user's preferences/interests.

Custom built DP: Fluid DP Synthesis: Tapping into user's selectedcategories and topics, a DP can be synthesized. DP is fluid and willchange over time based on user's preferences/interests. DP can besynthesized and tailored based on existing public informationrepositories (Freebase, dbpedia, Quora, . . . and private knowledge (ifaccessible).

Library of DP: Build a pre-defined set of topic-relevant DPs; Synthesizea library of DPs based on selected categories and relevant topics.Library will store all existing and new DPs for future processing. Ifon-the-fly synthesis causes performance problems.

Causality Graph for KD: Causality graph (CG) is vital for discoveringhidden topics that are important to connecting known topics. Hiddentopics discovered by CG are vital to discovering other importantrelevant topics. In particular, when a feature set of the referencetopics (topics, categories,user feedback) and its relevant topics cannot discover important topics for further knowledge discovery, minedhidden topics can be the answer. Hidden topics go beyond topics definedby the words or phrases or known relationships of reference topics.

Competency Queries: Competency questions/queries (CQ) aid in seeding aset of interesting questions to answer about the reference topic or therelevant topics that can be extracted. CQ are also important to seed adiscovery template to augment the topic at hand.

Competency Queries Extraction/Mining: CQ can be manually crafted by theuser/system. CQ can be automatically extracted or synthesized. Forexample, the knowledge discovery and augmentation system can querydatabases for questions relevant to topic and select the highest rankedquestions that history shows people care about. Quora is an example ofsuch database that can be mined to extract a set of CQ for a topic. Inan enterprise setting, to mine a set of CQ about a product, customer'sfeedback/queries/marketing data can be mined to synthesize a set of CQrelevant for a product. This CQ will serve as a seed to craft aDiscovery Pattern that will serve in augmentation of the relevant topic.

Causality Relationships: Relationships Characterizing; causing andleading relationships can be constructed based on mining a causalitygraph that is constructed beforehand. Involved Relationship: Same actorsinvolved in different topics in same timeline.

Topic Relationships: Topic Relationship Model is a mapping R such thatT1 R T2: Discovered topics and docs that connect T1 and T2; R completesthe knowledge graph that is relevant to the reference content. Example:Topics, Entities, Categories.

In accordance with this disclosure, the components, process steps,and/or data structures described herein may be implemented using varioustypes of operating systems, software development platforms, computingplatforms, computer programs, and/or general purpose machines. Inaddition, those of ordinary skill in the art will recognize that devicesof a less general purpose nature, or having limited resources mayrequire modification of an implementation of an illustrated embodimentwhich maybe done without departing from the scope and spirit of theinventive concepts disclosed herein. Where a method comprising a seriesof process steps is implemented by a computer or a machine and thoseprocess steps can be stored as a series of instructions readable by themachine, they may be stored on a tangible medium such as a computermemory device (e.g., ROM (Read Only Memory), PROM (Programmable ReadOnly Memory), EEPROM (Electrically Erasable Programmable Read OnlyMemory), FLASH Memory, Jump Drive, and the like), magnetic storagemedium (e.g., tape, magnetic disk drive, and the like), optical storagemedium (e.g., CD-ROM, DVD-ROM, paper card, paper tape and the like) andother types of memory.

The term “exemplary” is used exclusively herein to mean “serving as anexample, instance or illustration.” Any embodiment described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other embodiments.

In general, various information processing techniques and algorithms canbe used to provide the augmented data system with global and localcontext sensitive augmented content. In the following paragraphs certaindefinitions and representations of data flow models are presented anddiscussed without limitations on how each model may be implementedwhether by hardware, software, firmware, or any combination thereof.

Multilevel, e.g. global and local, context sensitive augmented contentwould increase productivity and enhance a user's experience whileviewing or interacting with data for the purpose of learning, reading,writing, drawing, browsing, searching, discovering, viewing images orany type of user interaction with digital data information whetherstructured or unstructured (e.g. financial, health, manufacturing, andcorporate data). The digital data information may be stored locally orremotely via a corporate server or in the cloud. Additionally private aswell as public sources of data may be used or selected by the user forthe ultimate personalized range of choices that may be used to furthernarrow down or expand the augmented content being presented.

Additionally, multilevel context sensitive augmented content wouldincrease productivity and enhance business intelligence for theenterprise by providing context sensitive augmented content that isgenerated by dynamically mining and analyzing structured andunstructured enterprise data and/or possibly leveraging structured andunstructured publically available data for further improving userexperience. In addition, multilevel context sensitive contentaugmentation filters provide the ability to dynamically mine data on thefly based on modification of a new input from a user. For example, a newinput from a user can be the selection of a new text or a portion of thereference content, or it can be a feedback provided such as elevatingthe priority or weight (e.g. like) or decreasing the priority or weight(e.g. dislike, delete, dismiss) of a single augmented content, acategory of augmented content, or a theme of augmented content.Furthermore, leveraging the history and/or user personal preferences,the multilevel context sensitive augmented content can be further intune with what the user would like to see or expects to see in theaugmented content being generated and presented.

In accordance with one embodiment, a feature of the multilevel contextsensitive augmented content is that the augmented content is generatedeither in the cloud or locally using sophisticated information retrievalalgorithms or using a set of heuristics so as to enable large-scale dataprocessing, information retrieval, and web mining. Knowing thatextracting a feature set from a web page is a problem that is known andvarious algorithms, methods, and research into various solutions havebeen made, this system can use existing research or methodologies toextract a feature set. Furthermore, this system employs a set ofheuristics and metrics that efficiently extract a set of features thatcharacterize the reference content at hand. These heuristics rely onembedded hints, metrics, metadata, or other embedded knowledge andinformation that can be extracted from the structure, url link, embeddedlinks, title of the document, or other types of data that may bedirectly or indirectly related to the reference content along withfeedback provided by the user.

In accordance with one embodiment, a feature of the multilevel contextsensitive augmented content is that the information retrieved andknowledge constructed can be saved and called upon in futureaugmentation tasks and sessions.

In accordance with one embodiment, a feature of the multilevel contextsensitive augmented content is that the augmented content is presentedthrough a translucent layer on top of the original content being viewedby the user. Hence a non-obtrusive content augmentation that is hiddenor made available whenever a user disables or enables the global andlocal context sensitive augmented content application. Relevantaugmented content are displayed on top of a translucent layer on top ofthe original content being viewed by the user. Hence, the augmentationsystem provides a less obtrusive and more efficient interaction,browsing and exploration experience.

In accordance with one embodiment, a multilevel corresponds to at leasttwo levels, a global level and a local level. A global and a localrelevant features of reference content, maybe defined as a globalrelevant feature corresponding to a feature or a theme common throughoutthe reference content, and a local relevant feature corresponding to afeature strongly related to a locality within the reference content. Onemethod of dynamically updating augmented content can be achieved byleveraging real-time user feedback, such as elevating priority ordismissing augmented content as being presented to the user. If anaugmented content's priority is elevated, its weight increases as wellas the metadata that describes this augmented content gets promotedwhich in turn updates existing augmentation filters as well asgenerating and presenting new augmentation content based on the newmetrics. For example, if an augmented content describing certain publicpolicy information is promoted, then that augmented content's priorityis increased, and the priorities of all augmented content that referencesome public policy, or government policy get increased. In addition, theaugmented content can be dynamically updated based on user interaction,e.g. selection and/or clicking, within the reference or augmentedcontent in real time. There are various means to implement the augmentedcontent presentation layers such as dials for global and local augmentedcontent, or a scroll-area of small windows for various augmentedcontent. Describing all these various means to implement the augmentedcontent presentation layer is not necessary to understand thisdisclosure. Furthermore, a person skilled in the art would understandand would be able to employ many different means to implement augmentedcontent presentation layers without departing from the spirit of thisdisclosure.

In accordance with one embodiment, while generating augmented contentmay result in a lot of data that cannot be shown on the display, thisdata can be stored in a deep queue. A deep queue means that there ismore augmented content (data) in the queue than what is displayed on thescreen. For example, not all mined augmented content can be displayedsimultaneously due to physical screen size limitations or the displaylayer size. A user can hover over the queue or press an arrow to scrollthrough the augmented content in the queue. In addition, it is importantto note that the augmented content being presented to the user maycomprise actual data, snap shot of the actual data, a processed portionof the actual data, or a link to the location where the actual data canbe retrieved.

Theme-based augmented content can further enhance a user's experience bypresenting a set of themes. In accordance with one embodiment, when theuser selects or deselects a theme, a new or updated augmented content ispresented to the user. An option to expedite augmentation and improvethe quality is to rely on the user's preferences and feedback. When theapplication is invoked, a set of categories/themes can be presented tothe user. These constitute metadata. By relying on the user choices ofthemes, augmentation can be enhanced and filtered. For example, aresearch paper that deals with AIDS virus would trigger a set of themessuch as Pharmaceuticals; Discrimination, etc . . . The user who isinterested in science and pharmacology but not in the social aspectsrelated to AIDS would deselect ‘Discrimination’. Thus, all augmentedcontent presented will be tailored to refer to categories that arerelated to science and other related aspects of the research. The themecan further be defined by a category or a set of related categories.This will serve to prune the augmented data and only present therelevant data that is of interest to the user and the task he iscarrying out at that moment.

Multilevel context sensitive augmented content application can beimplemented as a stand-alone application, on top of another application,or as an extension for applications, e.g. a browser extension. Inaccordance with one embodiment, further refinement or fine tuning ofvarious options for customization of augmentation system such asaggregating, mining, filtering, and presenting various aspect of data ormetadata can be performed dynamically in real-time. In addition, thecustomization of augmentation system maybe performed based on at leastone or more of a user's feedback, attributes, characteristics, theme,topics, and interests. Also when augmentation system presents a list oftags/categories, the user can provide feedback in the form orliking/disliking the tag. This is similar to promoting or dismissing anaugmented content. Therefore, in accordance with one embodiment, theaugmented content can be updated live. Furthermore, this user's feedbackwould also result in updating various subsystems such as the underlyingdata-mining, statistical computing algorithms, or machine-learningalgorithms or other information retrieval algorithms or heuristics.These updated subsystems are used to generate or create new signatures,metrics, or features which are based on user's feedback, e.g.liked/disliked tags, where the new signatures are used to generate newaugmented content or update the currently presented augmented content.

In accordance with one embodiment, a feature of a system for generatingand presenting multilevel context sensitive augmented content is theability to utilize online and offline mining and analytics foraugmentation. For example, mining and processing in real-time or inbatch mode and store data in a data store (local or remote) orpresenting real-time augmented content to the user. The stored data canbe used for future augmentation. Metadata and other relevant dataelements can also be annotated in real-time to capture user'spreferences and experiences. In addition, metadata and other relevantdata elements can be stored in a central repository to be leveraged forfuture augmentation of same or similar content. A brief description ofmetadata is that it is data that describes other data. For example:‘public health’ is a category that encompasses diseases. This higherlevel category ‘public health’ is a metadata for diseases.

In accordance with one embodiment, a multilevel context sensitiveaugmented content system uses at least two levels, a global level and alocal level. The following explains the difference between global andlocal augmented content. Global augmented content refers to augmenteddata that pertain to the overall document that the user is currentlybrowsing, exploring, or interacting with. A local augmented content canrefer to augmented content based on a particular piece, paragraph,sentence, word, image, icon, symbol, etc . . . of that document that theuser is currently browsing, exploring, or interacting with. Global andlocal augmented content are presented using a dynamic deep queue, andthe user can control the displaying of at least a portion of theaugmented content. Content sources for augmentation can be provided frommany sources. An example of such content sources includes but is notlimited to a user's own documents and data on desktop, web-content,social media sites, enterprise data-marts, and local and remote datastores, ontologies, other categorization, and/or semantic orrelationship graphs.

The multilevel context sensitive augmented content can be successfullyimplemented to augment a user's browsing experience as discussed above.In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content can besuccessfully implemented as an application for augmented user experience(UX). The system can increase productivity, provides augmenteddata-mining and data-exploration platform, augmented e-learning ande-research system, augmented desktop-based and mobile-based browsing,exploration, research, discovery, and learning platforms, dataaugmentation for better healthcare products and services, dataaugmentation for better educational products and services, augmentationsystem for better content management and relationship platform for bothenterprise and consumer applications, enhanced online-shopping researchand UX, enhanced marketing campaigns, an enhanced news access UX are butto name a few of application benefiting from a system for generating andpresenting multilevel context sensitive augmented content.

Semantic processing is the process of reasoning about the underlyingconcepts and expressing their relationships. In addition to variousaugmentation methods as described above, the following semantic basedtechniques can also be used in a system for generating and presentingaugmented content. In accordance with one embodiment, utilizing existingtags in public sources, utilizing batch-processed tags as a cloudapplication, semantic processing of selected content to generate a matchto an existing tag, semantic processing to generate augmented content onthe fly and utilizing user's feedback for promoting and dismissingaugmented content are but examples for methods to provide a betteruser-relevant augmented content. Generating augmented content on the flycan also be accomplished by using a feedback mechanism provided by theuser to enable mining and generating of new augmented data to bepresented to the user.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content is used toimprove the analytics of large data sets by leveraging pre-processeddata and already generated relationships. Given a content that is theresult of a statistical data mining and exploration functions on a smallor large amounts data—be it remote or local—extract the correlationmetrics and other signatures that demonstrate a meta-relationship andleverage it in other data-mining, analytics, and to generate augmentedcontent. For example: When a user presents some key words to a searchengine, the user gets a set of links that are related in addition tosome ads that could very well be related to the key words you haveentered or to some personal data known or extracted of the user. Thesepresented links and ads have gone through a huge amount of processingand computation in the cloud. By knowing that a relationship or ameta-relationship exists between the keywords, links, and may be othercontent pushed to the user like ads, the analytics operation canleverage them and extract, store, and leverage these signatures forfuture browsing or for presenting context sensitive augmented content.

In accordance with one embodiment, the content presented to the searchengine can be either parsed from the html or other format or interfaceproduced by a data provider. Or, it can be scanned through OCR if thedata format is encrypted. This ability to take a snap shot of a screenand analyzes and leverages its data and relationships empowers andsimplifies the augmentation and analytics processes and improves thethroughput since the signatures/correlation metrics extracted are aresult of processing a significantly smaller set of data. Therefore, theperformance gain of a system for generating and presenting multilevelcontext sensitive augmented content is orders of magnitude compared tomining massive data sets in the cloud.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content presents theaugmented content along with the reference content using two or moredifferent presentation layers displayed using the same display screen.In addition, the system provides the ability to customize the generationof augmented data in situ (in place) while working on original orreference content, where the augmented data can be displayed on see thrupresentation layers so as not to obscure the original or referencecontent and to maximize use of the display screen, and/or the displayingarea.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content utilizesdynamic updates of displayed augmented content using presentation layerswhile a user views and manipulates reference content displayed usinganother presentation layer. It is preferable to use a translucentpresentation layer for the augmented content presentation layer that islocated on top of the displayed reference content so that the user caneasily manipulate or interact with the reference content whilesimultaneously viewing the dynamically updated augmented content. As canbe easily appreciated by person skilled in the art that displayingrelevant augmented data in a separate tab or page would result in lossof context relationship and provides a less efficient and less friendlyuser experience. Similarly, displaying the augmented content on thesidebars is possible as well. However it consumes screen space andclutters displaying of the reference content. Therefore, the ability tokeep the reference content accessible to the user while displaying theaugmented data on top of the original content provides a much smootherand efficient user experience. Furthermore, the user can easily hide,size, move, or display the augmented content without affecting thereference content.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content enables a userthe ability to associate any of the augmented content with the referencecontent or an attribute of the reference content source using one ormore types of metadata. The system enables the user to save theassociated metadata for future use or sessions. For example, theassociation of metadata can be accomplished by embedding a link in thetext, by associating a link with a text, or by associating any data ormetadata with the reference content or any part of the referencecontent. Moreover, the user has the ability to specify a category ormore as a source or criterion of augmentation. The user can also defineassociation rules that join a group of attributes, categories, and othermetrics together to provide a richer input to aid the augmentationsystem to generate more relevant augmentation content. For example, anenterprise sales projection document can always be augmented with anydata source or data documents that generated the projection. Thecriterion is a category that says source sales data and not necessarilythe exact data documents. The sales data can be extracted automaticallyby the augmentation system. Utilizing selected or provided categories ofinterest, the augmentation system can carry out an updating procedurefor any associated data or metadata for any other reference content.Furthermore, the augmented content is displayed using translucent layersso that the user always sees and has access to the original or referencecontent. The user is able to access, browse, move, select, hide, tap,scroll, or interact with the reference or augmented content while thesystem dynamically generates and displays an updated augmented contentusing the augmentation presentation layer. It is noted that the userinteraction with the reference or augmented content can result in havinga new reference content that the user wishes to interact with, hence, anew augmented content is generated and displayed. The system keeps trackof and saves certain information regarding this nested augmentationlevel. The system provides the user the ability to switch back and forthbetween various nested augmentation levels as well as saving or sharingthe augmentation filters or settings used for a particular session.

In accordance with one embodiment, further enhancement of the userexperience is achieved by enabling the user to change the skin (or lookof a user interface UI) of the augmentation system. For example, thesame components of a UI (buttons, options, data) can be displayed on thescreen in a variety of ways. Usually, a library of templates and coloroptions can be provided to allow the user to customize the display ofthe augmented content presented by the application. In addition, theglobal augmented content and local augmented content can be displayedusing one or more different regions of the screen, or displaying theglobal and local links to the augmented content in two concentriccircles around the reference content. The enhancement of the userexperience is achieved by enabling the user to choose the most efficientway to utilize and display the augmented content.

In accordance with one embodiment, user selectable skins can also beused to cover or hide pushed content that may exist or embedded in thereference content being viewed. User selectable areas of a skin can beused to enable the display of user selected content such as images oraugmented content, or pushed content such as advertisement. For example,an ad for tickets to a local concert when the user is browsing aspecific artist, or an ad for a book that relates to a global or localaugmented content of the user reference or currently viewed augmentedcontent, or any other monetization mechanism based on the augmentationprocess. The enhancement of the user experience includes a nestedmultilevel context sensitive augmented content where the augmentedcontent presented to the user can be further enhanced as a function ofthe various nested levels. The augmented content is presented whilekeeping track of the current content being viewed in relationship to theoriginal content that the user started with and all levels in between.This provides a hierarchical augmentation system that enables the userto access and build nested levels of augmentation.

In accordance with one embodiment, the user interface, or UI, for asystem for generating and presenting multilevel context sensitiveaugmented content can be launched or started automatically, and canreside in the background and stays hidden from view until the userinvokes a predefined programming function to enable the UIfunctionality. For example, a single tap, hot-key, function-key, agesture, or a multiple or a combination of actions acted upon a contentwould cause the transparent augmentation layer to be shown with theaugmented content and in accordance with user preferences, such as tags,skins, themes, etc . . . Selecting content presents or updates theaugmented content already presented. Visiting an augmentation linkresults in completely or partially (split screen) covering the referencecontent or original layer comprising the original content. The UIprovides the user the ability to navigate nested augmented content orjump back to reference or original content.

In accordance with one embodiment, additional system and UI features canfurther be used to increase the overall efficiency and provide a betteruser experience. For example, saving the augmented content metrics inuser history, and using history to enhance and/or tailor analytics andaugmentation as would be more relevant to each individual user or groupof users such as in corporate environment. Metrics here refer to thegenerated signatures as mentioned above. Also, it refers to anyannotations that are provided by the user such as priority,liking/promoting an augmented content or dismissing it. This can bestored for future sessions as well as using the augmented contentpromotion and dismissal to enhance augmentation in real time. Usingskins that cover an undesirable part of the screen, e.g. side columnswhere ads are pushed. The skin may be used for further customization ofthe viewed screen and potentially could be monetized and leveraged topresent relevant augmented content that is paid for by the user, such asads for objects, e.g. books, related to the content of a referencearticle.

In accordance with one embodiment, a system for generating andpresenting multilevel context sensitive augmented content providesdynamic user-guided and customized context-sensitive data augmentationto facilitate learning, exploration and knowledge discovery. The systemprovides simultaneous interaction with the augmentation layer and thecontent layer. The system generates augmentation data based onuser-defined metrics and filters such as themes, categories of interest,document content and/or part of it. The generated data is not a rigidaugmented content. The generated augmented content is any data, concept,and relationships that are presented as a result of the data mining andprocessing of the original content and the user-defined metrics andfilters.

In accordance with one embodiment, the system utilizes dynamic andinteractive methods to successively refine and tailor the augmentedcontent based on a user's guidelines, filters, and metrics. The systemrelies on a variety of sources for content augmentation by accessing anyonline or offline databases, crowd-sourced databases, or open databases.Furthermore, over time, a custom built graph of concepts andrelationships can be built between different pieces of data as they areprocessed and augmented based on the user's filters and metrics toimprove the performance of the system and the User Experience. Thesystem provides a context-sensitive hierarchical augmentation frameworkfor deeper and expansive exploration and knowledge discovery. The systemenables construction of a customized graph of data, concepts, andrelationships based on the filters and metrics provided even in theabsence of content. Content can be generated on the fly for furtherexploration.

In accordance with one embodiment, the system enables sharing ofaugmented data and the associated metrics that generated them. Thisenables richer knowledge discovery by further refining a user'saugmented data based on other users' augmented content. This is usefulfor collaborative research and knowledge discovery. The system can belaunched from offline and online documents or reference content togenerate the augmentation content, data and graph of relationshipsamongst the concepts represented by the augmented content.

In accordance with one embodiment, the system provides a UI to displayand manipulate reference content and augmented content concurrently,dynamically, and interactively. The system provides one ore moretranslucent layers on top of the reference content to show the augmentedcontent. Translucent layers facilitate displaying the reference contentas well as the augmented content. Translucent layers can fully orpartially cover the original content. Augmentation layers can be hidden,minimized ((shown as icon), or moved around on the display screen tofacilitate easier display and interaction with the reference content.The system enables the user to manipulate and control a set of displaylayers (reference content layer, and/or augmentation display layers) ina very flexible fashion such that the user can size up, down, move,show, hide any of those display layers. The system provides anintuitive, rich, and friendly UX for data exploration and knowledgediscovery on small and large display screens. In particular, displayingof the augmented content concurrently and interactively on the originalcontent empowers the user to use this system on smart phones, tablets,and any other display. Furthermore, the system provides means to insertadditional content on the augmentation layers based on analytics on theaugmented content and the original content.

A block diagram of an Augmentation System 100 is shown in FIG. 15. AReference Content 105 corresponds to any electronic document or web pagethat a user wants to invoke the Augmentation System 100 to get AugmentedContent 190. The Reference Content 105 can be stored locally in a memorysubsystem of an electronic device, a memory subsystem of a displayscreen device, or is accessed from a remote location via a wired orwireless communication system. The communication system could use theinternet, a cloud, a data store, a computing device, server or adatabase via a wired or wireless networking link. The augmented contentis a generated content by the Augmentation System 100 based on theReference Content 105 using a set of features, filters, and categorieswhich are produced by at least one of an Extract Features 120, ExtractCategories 125, and Update Categories 137 subsystems as shown in FIG.15.

A Local Content 110 is a selected portion of the Reference Content 105which the user wishes to get more specific augmentation about, or thatis a portion of the Reference Content 105 that the user is interactingwith. Furthermore, the Local Content 110 may also be automaticallyselected, tagged, managed, or generated by the Augmentation System 100,e.g. based on a displayed portion of the Reference Content 105 or a userinteraction with a portion of the Reference Content 105. Furthermore,the presentation and/or the displaying of the Augmented Content 190 ismanaged using Manage RAC 145 (RAC refers to Relevant Augmented Content)to control a Display Queue 165 and Display RAC 170.

The Augmentation System 100 generates Augmented Content 190 byfacilitating the construction of a user-customized network of concepts,objects and relationships that serve to augment the Reference Content105 at hand for the purpose of knowledge discovery, learning, and aricher user experience in browsing and/or interacting with datainformation. This Augmentation System 100 generates any one of a networkof concepts, a network of objects, and a network of relationships usingone or more of a set of features, a set of filters, and a set ofcategories. Each of the set of features, the set of filters, and the setof categories can be customized and tailored based on the user'sinterests and input. The constructed network can be saved and furtheraugmented over time for richer and more efficient user experience.

The Extract Features 120 subsystem extracts a set of features from theReference Content 105. General Features 117 can provide a set offeatures that can be updated and tailored overtime to at least one of aspecific user, specific project, specific objective, and specificsubject. Extract Features 120 generates a set of filters that denotesthe desired concepts for augmentation. For example, these concepts couldbe names of people, history, events, topics, or other metadata. Thesedata are either computed on the fly or pre-computed and stored locallyor remotely for current or subsequent augmentation sessions. Thisextraction process is based on embedded data in at least one of theReference Content 105, in a linked content to the Reference Content 105,metadata of the Reference Content 105, e.g. a title of the ReferenceContent 105, linked content to the Local Content 110, and semanticinformation that are either associated with the Reference Content 105 orthat can be extracted/aggregated from the Reference Content 105. Otherdata that can be extracted or inferred can be further used forconstructing a more meaningful feature set by utilizing a variety ofinformation retrieval, extraction, and inference algorithms and methods.There is large body of work on feature extraction that utilizes thecloud as well as other large-scale solutions. These approaches can beleveraged by the Extract Features 120 along with flexible and efficientalgorithms to generate a set of features on the fly based on the metricsand signatures mentioned earlier. Furthermore, any part of theAugmentation System 100 can be run remotely on a server or in theclouds, or it can be run locally on the host device.

The Extract Categories 125 function uses a set of categories or topicsthat are extracted based on the data that can be associated or extractedfrom the Reference Content 105. This data can be either metadata or anyother related data to the Reference Content 105. The Extract Categories125 extracts a set of categories from the Reference Content 105 and itsassociated links and data. Also, the system utilizes any embeddedcategories or metadata that are either embedded in the link or attachedto the Reference Content 105. The extracted categories can also describemetadata about the topic at hand. For example, if the reference contentis an article about AIDS, there are many categories that can augmentdata about AIDS. For example, a set of categories can be: History ofAIDS, Science of AIDS, Social Impact of AIDS, Symptoms of AIDS, etc . .. A user may only be interested in the science of AIDS, so a user willinteract with the presented categories, e.g. by deselecting allcategories that are not related to science, and this will impact the setof features that are used in augmenting the Reference Content 105. Otherdata that can be extracted or inferred can be further used forconstructing a more meaningful category set by utilizing a variety ofinformation retrieval, extraction, and inference algorithms and methods.In addition, a General Categories 115, as shown in FIG. 15, is a set ofdefault categories that the Update Categories 137 processes to reflectthe user's interests. For example, the General Categories 115 can beBusiness, Politics, Education, Research, Health, Technology, etc . . .The Update Categories 137 may use this optional input from the user tobias the augmentation to the categories of interest. This optional inputcan be stored and updated over time.

The interaction of a user with the Augmented Content 190 maybeaccomplished in a variety of ways. For example, the user may select oneor more of the presented categories for removal, selection, decreasingpriority, and increasing priority. The user may also define, modify, orinteract with an association rule to aid Extract Features 120 togenerate a more useful set of filters for better augmented content. Theassociation rule can leverage, use, or join one or more categories,features, filters, or concepts to (i) generate a new set of features,filters, categories, or Augmented Content 190, and (ii) to modify one ormore of the set of features, filters, or categories which are being usedto generate the Augmented Content 190. Based on the General Categories115 and user's interaction, further categorization and featureextraction will be biased towards the user's interaction or input. Thisis an optional input that is used to customize the Augmented Content 190based on a user's needs, the user's interaction with Augmented Content190, or to aid the Augmentation System 100 to provide more relevantAugmented Content 190 for a specific purpose. Upon a user's interactionwith the Augmented Content 190, an Update 130 function enables theuser's input to be considered by Update Categories 137, e.g. a user maychoose to delete some of the default/general categories that are not ofinterest or to elevate the priorities of some of those categories. Whendeleting categories, the Update Categories 137 will reduce the weight ofthe features that are related to those categories. When categories areelevated in priority, the Update Categories 137 increases the weightgiven to those features that are related to those categories. Thus,affecting and updating the Augmented Content 190 presented to the user.

An Update Filters 150 is used to indicate a user's preference for afeature or automatic feedback based on user's interaction with theAugmented Content 190. For example, when one or more of the ReferenceContent 105, Local Content 110, and Augmented Content 190 get updated orinteracted with by a user, then more clues and feedback can be gatheredfrom the updated list or the user's interaction as to revise thefeatures and categories that are of interest to the user in real time.However, the user may choose not to update the features and categories,and the Augmentation System 100 provides the user the ability to controlhow and when the Augmented Content 190 is generated and/or updated.

An Update Features & Categories 135 subsystem receives a first set offeatures from the Extract Features 120 subsystem, a first set ofcategories from the Extract Categories 125, and/or an updated set ofcategories from the Update Categories 137, and/or an Update Filters 150.Update Features & Categories 135 manages and controls the updating ofthe actual features and categories sets including any decision makingbased on the user input or interaction. The Update Features & Categories135 may communicate with any one of Extract Features 120, ExtractCategories 125, and Update Categories 137 to generate more features andcategories based on a variety of parameters including the user'spreferences. Furthermore, Update Features & Categories 135 also handlesupdating relationships and cleaning up for those features and categoriesthat were updated by the user.

A Compile RAC 140 subsystem receives a set categories and a set featuresfrom the Update Features & Categories 135 subsystem. Compile RAC 140includes a variety of functions and algorithms such as machine-learning,data mining and extraction, web crawling, data-mart accessing,extraction and processing functions, and other intelligent algorithmsand approaches are used to compile a set of relevant augmented contentor pages (RACs) based on at least one of the Reference Content 105,Local Content 110, and the interest of the user. Managed RAC 145subsystem is the controller that manages the presentation of theAugmented Content 190 via a Display Queue 165 and Display RAC 170. TheAugmentation System 100 listens to inputs from the user and manages thegeneration of the Augmented Content 190. The Managed RAC 145 subsystemgenerates three outputs taking into consideration a user's feedback orinput. The Managed RAC 145 subsystem generates and controls thecommunication of the generated Augmented Content 190 using Display Queue165 and Display RAC 170. In addition, Managed RAC 145 generates anupdate request to Update RAC 155 for any necessary update to the DisplayQueue 165 based on a user's interaction or input. The Display Queue 165displays in a desired skin at least a portion of the queue of RACs sothat the user can browse through them and select some to view. TheDisplay Queue 165 displays a link, a summary, or a portion of thecompiled relevant content or pages. Upon selection or interaction by auser with one of the displayed RACs, the Display RAC 170 retrieves therespective relevant page RAC and displays at least a portion of it. TheDisplay RAC 170 subsystem manages and controls the displaying of theAugmented Content 190 using the display screen. Display RAC 170 can useone or more display layers on top of the Reference Content 105 or LocalContent 110 via translucent display layers as discussed in previousparagraphs.

A block diagram of a Hierarchical Augmentation System 200 is shown inFIG. 16. This hierarchical augmentation or nested augmentationcapability enables a user to augment any content that is the result ofdata augmentation at any level of browsing or exploration. For example,given that the Augmentation System 210 generates a list of RACs, theuser may select any one of the RACs or a group of RACs to invoke theaugmentation system on and to generate another level of augmentation.The Augmentation System 200 allows the user to go back and forth in thehierarchical graph to browse any particular content at any level, be ita reference or augmented content. For example, the Augmentation System200 provides augmented content at Process 1 220, which is the firstinvocation of the augmentation system on reference content, a user mayelect to augment one or more of the augmented content of Process 1 220.The Augmentation System 200 uses the elected content to be augmentedfrom Process 1 220 as an input or reference content to Process 2 230 foraugmentation. Process 2 230, which is considered the second invocationof the augmentation system on a reference content, generates in turnaugmented content which the user can further refine or interact with,and so on for Process k 340, Process (n−1) 350, and Process (n) 360.Multilevel nesting or hierarchical augmentation is not limited to aspecific number of levels. Of course certain hardware or softwarelimitations or a particular application may dictate the use of aspecific number of levels. However, this is an option that can be usedto various extents as part of the customization of Augmentation System200 for any particular usage.

A block diagram of a Hierarchical Augmentation System 300 is shown inFIG. 17. This hierarchical augmentation or nested augmentationcapability comprises the same capabilities as the HierarchicalAugmentation System 200 is shown in FIG. 16 and includes an AugmentationSystem Control 390 subsystem that is communicating Augmented andReference contents AR-325, AR-335, AR-345, AR355, and AR-365 withProcess 1 320, Process 2 330, Process k 340, Process (n−1) 350, andProcess (n) 360, respectively. As described above Process 1 320corresponds to a first level instance of Augmented System 100, andProcess (n) corresponds to an n-th level instance of Augmented System100. Given that each of the nested augmented systems may generatedifferent augmentation content for each hierarchical level at least dueto variations in user input or the reference content corresponding tothe hierarchical level, the Augmentation System Control 390 may receiveone or more of the generated augmented content of each hierarchicallevel, a copy of the set of filters, a copy of the set of features, anda copy of the set of categories. The Augmentation System Control 390 canfurther run sophisticated statistics, analytics and algorithms toextract new features or generate new filters or categories. Furthermore,the Augmentation System Control 390 may receive user input to controlwhat type of analysis or augmentation the user expects the HierarchicalAugmentation System 300 to provide or keep track of nested contents thatthe user is interacting with, viewing, or manipulating at various levelsof hierarchy.

A block diagram of an Augmentation System 400 using a Display Control420 subsystem is shown in FIG. 8. The Augmentation System 410 isessentially the same as any one of the Augmentation System 100,Augmentation System 200 and Augmentation System 300 as shown in FIG. 5,FIG. 6, and FIG. 7 respectively. The Display Control 420 subsystemcontrols the displaying of various elements such as Augmented Content450 and Reference Content 440, which are output of the AugmentationSystem 410. In addition, the Display Control 420 receives input controlfrom Augmentation Display 430 subsystem and/or from a user interactingwith the Augmentation Display 430 or one or more display layersdisplayed using the Augmentation Display 430. Based on the AugmentedContent 450 generated from Augmentation System 410, Display Control 420generates and/or controls different display layers, widgets, icons, andother knobs which are utilized to show, control, or manipulate any oneof the Augmented Content 450 and Reference Content 440. Furthermore,Display Control 420 provides means for the user to interact with any oneof the Reference Content 440 or the Augmented Content 450.

In accordance with one embodiment, the Augmentation System 410, theDisplay Control 420, and Augmentation Display 430 are elements of thesame physical electronic system such as a mobile device. The user canmanipulate any one of the Augmented Content 450, Reference Content 440,and how each is displayed onto the Augmentation Display 430. Inaddition, a user interface (UI) may be used to further aid the user tomanipulate or interact with any one of the Reference Content 440 and theAugmented Content 450 and the displaying of such content. Furthermore,the UI can provide an easy mechanism for a user to interact with thecategories, widgets, buttons, and any other option that is presented forthe user to engage with the Augmentation System 410.

In accordance with one embodiment, the Augmentation System 410, and theDisplay Control 420 are elements of a first electronic device that isseparate from a second electronic device comprising the AugmentationDisplay 430, wherein the first and second electronic devices communicatethe Reference Content 440 and the Augmented Content 450 back and forthbased on the Augmentation System 410 and/or a user interaction with anyone of Reference Content 440 and Augmented Content 450.

In accordance with one embodiment, the Augmentation Display 430, and theDisplay Control 420 are elements of a first electronic device that isseparate from a second electronic device comprising the AugmentationSystem 410, wherein the first and second electronic devices communicatethe Reference Content 440 and the Augmented Content 450 back and forthbased on the Augmentation System 410 and/or a user interaction with anyone of Reference Content 440 and Augmented Content 450.

In accordance with one embodiment, a system for extraction andgeneration of relevant augmented content extraction is shown in FIG. 9.The Relevant Augmented Content Extraction 500 is tasked with building aset of features and categories that any one of the Augmentation System100, Augmentation System 200, Augmentation System 300 and AugmentationSystem 400 can utilize to generate augmented content. Reference Content510 is similar to Reference Content 105, and Local Content 520 issimilar to Local Content 110. Categories 530 is a subsystem which isresponsible for constructing a list of categories that captures or isresponsive to the user's inputs and preferences, a set of extractedcategories from Reference Content 510 and Local Content 520, and a setof customized categories associated with the user. Features and Metrics525 is a subsystem which generates a set of features, a set ofsignatures, and/or a set of metrics each of which is either dynamicallygenerated or pre-computed and stored. Features and Metrics 525 deliversthese sets of features to an Extract Features 540 subsystem. Inaddition, the Extract Features 540 receives input from Reference Content510, Local Content 520, Features and Metrics 525, and Categories 530.Extract Features 540 delivers a set of features, a set of signatures,and a set of metrics to Compile RAC 550 subsystem, which in turnutilizes one or more of those sets to compile from the internet, a localdata store, or any other data repository (public or private) a set ofdata elements corresponding to the relevant augmented content to bepresented to a user. A Relevant Augmented Content 560 subsystem receivesthe set of data elements and prioritizes the relevant augmented contentin accordance with a user preference or interaction, the set of featuresor filters, the set of signatures, and the set of metrics to generate acustomized augmented content for the user.

In accordance with one embodiment, a simplified block diagram of asystem for a relevant augmented content extraction is shown in FIG. 10.The Relevant Augmented Content Extraction 600 can be used as a part ofan augmentation system such as Augmentation System 100, AugmentationSystem 200, Augmentation System 300 and Augmentation System 400 each ofwhich has been described above. The Extract Relevant Features 600 isutilized to compile a set of features, using Compile RAC 650, to be usedby an augmentation system to generate augmented content. ExtractCandidate 618 processes at least a portion of a Reference Content 608,and receives other user-provided input to extract or generate one ormore set of filters and features. Features 620 uses the one or more setof filters and features to organize, build, compile or store auser-customized network of features, concepts, objects and theirrelationships. Features 620 serve to provide a better extraction or afocused extraction of a user's relevant set of features that can providea faster convergence on what the user is interested to see or would wantto see regarding the Reference Content 608. In addition, this provides abetter value add augmented content for the purpose of knowledgediscovery, learning, and a richer user experience in browsing and/orinteracting with data information. Furthermore, Features 620 can learn,save and further refine the user-customized network of features,concepts, objects and their relationships over time for richer and moreefficient user experience. Similarly, Categories 630 uses the one ormore set of filters and features generated by Extract Candidate 618 toorganize, build, compile or store a user-customized network ofcategories and their relationships. Categories 630 can learn, save andfurther refine the user-customized network of categories and theirrelationships over time for richer and more efficient user experience.

Metrics 640 is a system that can provide user influenced metricsinformation to Compile RAC 650. Metrics 640 uses the one or more set offilters and features generated by Extract Candidate 618 to organize,build, compile or store a user-customized network of metrics which canbe user defined or system's default. For example, Metrics 640 can usedate or time as a metric that can be used to further narrow and focus onthe relevance of the augmented content to the user or to the ReferenceContent 608. Another example is to use a source or a group of sources toaid Compile RAC 650 to limit or expand its compilation and generation ofrelevant augmented content. Metrics 640 can learn, save and furtherrefine the user-customized network of metrics and their relationshipsover time for richer and more efficient user experience. Metrics 640 canreceive real time information from the user or other part of anaugmentation system, and provides an update in real time to Compile RAC650.

Compile RAC 650 is used to compile the networks of features, categoriesand metrics received from Features 620, Categories 630, and Metrics 640to generate and prioritize a focused set of relevant augmented content(RAC) that captures the properties and/or attributes of ReferenceContent 608 and reflects the user's rules, interests, preferences, andattributes. This focused set of relevant augmented content (RAC) is tobe used by an augmentation system to deliver or present a concise andhighly relevant augmented content to the user. Compile RAC 650 is usedto resolve any conflicts that may exist between any of the networks offeatures, categories and metrics. Compile RAC 650 also provides anddetermines the priority of the final list of RACs to be delivered orpresented to the user. Compile RAC 650 can also receive, generate ormodify an association rule which can be used to leverage, or joins oneor more categories, features, filters, concepts, or metrics to (i)generate a new set of features, filters, categories, or relevantaugmented content, and (ii) modify one or more of the set of features,filters, or categories which are being used to generate the relevantaugmented content.

In accordance with one embodiment, an augmentation system can useDisplay Layers and Controls 700 as shown in FIG. 11 to display thegenerated Augmented Content 760, Global Augmented Content Queue 720,Global Augmented Content Queue 770, and the Reference Content 750. Forexample, this can be one instantiation of the data presentationmechanism of an augmentation system as described above, e.g AugmentationSystem 100. The user can change the look and feel (skin) of the DisplayLayers and Controls 700 using any number of skins (look and feeloptions). The Global Augmented Content Queue 720 corresponds to adisplayed part of a relevant augmented content (RAC) generated by theaugmentation system. The user can browse and scroll through this queueto select a relevant augmented content of interest. Local AugmentedContent Queue 770 refers to the relevant augmentation results that arerelated to the part of Reference Content 750 that the user hasinteracted with or is being displayed via Display Screen 710, and whichis referred to as Local Content. Display Layers and Controls 700 canmanage the display of the Global Augmented Content Queue 720 and LocalAugmented Content Queue 770 in various ways, such as the location of thedisplay of the queues as well as the portion of any one of the queuesthat is being displayed using Display Screen 710. For example, the usercan choose that only the Global Augmented Content Queue 720 isdisplayed, thus Display Layers and Controls 700 will manage to displaythe portion of RACs of the Global Augmented Content Queue 720 that maybeaccommodated onto the Display Screen 710. Similarly, the user may chooseto emphasize the Local Augmented Content Queue 720 and thus the DisplayLayers and Controls 700 will manage that as well. The Reference Content750 refers to the content being browsed and explored for furtheraugmentation. Display Screen 710 corresponds to a display screen thatmay be physically collocated within the same device where theaugmentation system is being used, or it can be part of a separateelectronic device. Augmented Content 760 is displayed using one or moredisplay layers, and is the augmentation content that the user chooses toview. An icon Promote 740 is used to highlight, select, or promote aspecific RAC. Promote 740 provides a mechanism for the user to interactwith any of the RACs of Global Augmented Content Queue 720 and LocalAugmented Content 770 by elevating the priorities of a RACs. Similarly,a demote icon (not shown) can be used by the user to remove or dismiss aRAC or a group of RACs entirely if the user is not interested in them.

A Simulated Display 800 is a use case scenario of the Display Layers andControls 700 and any one of the augmentation systems described earlieras shown in FIG. 12. This Simulated Display 800 presents an example of auser reading an article about AIDS as shown in Reference Content 810.The user then invokes the Augmentation System to augment ReferenceContent 810. Based on the categories presented to the user, the userselects categories that are related to AIDS Research and Science.Categories Related to AIDS RACs 820 shows part of the globalaugmentation deep queue that the system generated in response to theuser's interest in AIDS, Science, and Research.

A Simulated Display 900 is a use case scenario of the Display Layers andControls 700 and any one of the augmentation systems described earlieras shown in FIG. 13. This Simulated Display 900 presents an example of auser reading an article about AIDS as shown in Reference Content 910.Selected Content 920 shows an example of selecting part of the ReferenceContent 910. The user then invokes the Augmentation System to augmentReference Content 910 and Selected Content 920. Based on the user'schoice of categories related to Science and Research of AIDS, and thesystem's extracted categories and features RACs 905 shows part of theglobal augmentation deep queue that the system generated. Africa/IndiaAIDS RACs 930 shows part of the local augmentation deep queue that theAugmentation System generates in response to the user's selection ofpart of Reference Content 910. Augmented Display Layer 940 is an exampleof displaying of Africa/India AIDS RACs 930. Augmented Display Layer 940shows a RAC (HIV and AIDS) in the local augmentation queue that theelected to view.

An example knowledge discovery system 100, as shown in FIG. 15, includesa KDP system 195 that is coupled to multiple users, e.g. User A 101,User B 102, and/or User C 103. The KDP system 195 communicates viaAccess Channel 190 with each user by providing discovered knowledge oraugmented data and/or data associated with preferences or knowledgediscovery request or data augmentation request. In addition, each usermay also have a direct connection via communication network 175 or othercommunication means which may include wired or wireless communicationdevices. The KDP system 195 also includes Memory 185 to store datainformation and code associated with a KDP user, Share A-B 131, Share C132 and Discovery & Tailor Knowledge 121. User A 101 can access, use,manipulate, and display online content or private content and augmentedcontent received from the KDP system 195. User A 101 can include anytype of computer system or mobile device that allows its operator tointeract, transmit and receive data information.

An example of sharing a link or content between User A 101 and the KDPsystem 195 would occur via Access Channel 190. For example, using asocial network would be simply to copy the content and share it or toprovide a link to where the content is or the source of the contentwhere it can be accessed. Simply put when a link, e.g. a/v media orarticle, is shared then the end user cannot change the content.

An example of a Knowledge Discovery system, as shown in FIG. 15, is usedby one or more users to explore and share augmented content. Each userwould receive a shared augmented content that is locally enriched andaugmented by the KDP system 195 as per the receiving user's preferences,specific parameters, and/or geographical location. In addition thereceiving user is able to interact with the shared augmented contentusing the KDP system 195. This means adding more relevant content thatthe system can discover based on the receiving user's input orpreferences. For example, an original author may dismiss certainaspects/topics from an augmented content while another user may considerthose dismissed aspects/topics important. Thus, the KDP system 195having access to the other user's preferences would be able toregenerate the shared augmented content using his/hers preferences orprofile. Moreover, the KDP system 195 enables each user to enrich andaugment any shared content automatically and/or based on certainprogrammable parameters.

Another example of KDP system 195 is a recursive augmentation, where anoriginal author tackled one side of the knowledge and shared it withanother user who receive not only what the original author has sharedbut also a customized augmentation based on the receiving userpreferences where the KDP system 195 can automatically process andgenerate augmented content of the original author's shared content basedon the interest of the receiving user. This can allow the receiving userto dig deeper or expand the knowledge discovery. Furthermore, thereceiving user can highlight, augment or add his comments to anaugmented content, e.g. annotating or adding an external link to acontent such as a video/audio or a link to an article not discovered bythe system Ultimately, every user can become an investigative reporter,and each successive user who receives a shared augmented content canbuild on what he/she received from the perspective of his/her owninterests. A KDP system 195 is a smart knowledge discovery system thatenables democratization of knowledge.

Another example of sharing augmented content using a KDP system 195 isthat the shared augmented content or knowledge can embody all or part ofthe augmentation parameters used by the original author e.g. User A 101.A receiving user, e.g. User B 102, using the shared augmented contentwould be able to use the KDP system 195 to use this shared knowledge asthe seed for him to build on using his own preferences, parametersand/or specific interests as well as many other possible seed info foradditional augmentation as described above.

FIG. 16 presents an example data flow for User A 101, e.g. user A isreading an article/content. The user A 101 invokes the knowledgediscovery platform 110 on the desired content to further explore anddiscover relevant knowledge. The user A 101 can interact with theautomatically discovered knowledge. User A 101 tailors the discoveredknowledge by promoting certain content or dimension of knowledge, ordemotes others based on the task being accomplished or the interests ofuser A 101. User A 101 sets a policy in Knowledge Sharing andBroadcasting 130 for sharing the content with one or more end users 140or the public at large. The user A 101 can share or broadcast thediscovered knowledge represented by the knowledge graph and otherrelated augmented data that is automatically updated based on policy orinterests that is known to the KDP system 195 for each end users 140.Moreover, each user of the end users 140 can further augment or interactwith the shared content using the KDP system 195 which can provideselective feedback or additional content to the original user A 101 andto each end users 140 based on the collective content augmentation ofall users or a selective group of users as may be determined by user A101 sharing policy in Knowledge Sharing and Broadcasting 130 or asdetermined by each user of end users 140 own sharing policy 230 as shownin FIG. 17 and further explained below.

The KDP system 195 comprises Knowledge Discovery Tailoring andAnnotation 120 which receives the augmented data 200 and knowledgediscovered by the Knowledge Discovery System 100. FIG. 17 is data flowdiagram for the Knowledge Discovery Tailoring and Annotation 120. In theprocess of knowledge discovery, the KDP system 195 may discover muchrelevant content to the content in hand. The user A 101 may choose toemphasize certain aspects of the knowledge and/or dismiss others inAugment Topics 210. The user A 101 interacts with the KDP system 195 totailor the resultant knowledge. The user A 101 can further manuallyannotate the shared content in Annotate Data/Knowledge 220, e.g. to addhis/her remarks. The user A 101 can set the sharing policy in Sharingand Modification Policies 230, e.g. such as who can view it, who canedit it, and who can share it, etc. The user A 101 shares his discoveredview 240 with his circles of connections or broadcast it to the public.

FIG. 18 shows an example of Knowledge Discovery sharing policy KnowledgeSharing and Broadcasting 130. The augmented data 300 corresponds to theknowledge discovered or shared from a user A 101 of the knowledgediscovery system 100. The user receives and views the shared link 310 ofdiscovered knowledge. For example, the shared link can be transmittedusing any means for communication e.g. via email, text, social network,etc . . . Or, via some notification in the knowledge discovery platform110. The user shares the knowledge based on his sharing policy 320. Theknowledge discovery system 100 checks the membership of the user viewingthe shared link 310. If the user wishes to interact with the sharedcontent e.g. story, then the user is given an option to become a memberof the knowledge discovery system 100 and create his own profile orpreferences and thus be able to further refine or generate customizedaugmented content. The user can edit the knowledge discovered, tailor,or annotate and the shared knowledge 350. The user shares the tailoredknowledge 360 in a similar fashion as was described above. This processrepeats and could go viral in sharing and tailoring of knowledge.Moreover, the KDP system 195 can provide selective feedback oradditional content to the original user A 101 and/or to any user or agroup of users of end users 140 based on the collective contentaugmentation of end users 140 or a selective group of users as may bedetermined by user A 101 sharing policy 130 or as determined by eachuser of end users 140 own sharing policy 320. The KDP system 195 canexponentially grow the discovered knowledge as more users interact withthe original shared content.

The KDP system 195 can benefit not only the original user A 101 throughselective feedback of augmentation parameters or uses of his originallyshared content, but also the richness of the augmented content to thecommunity of end user 140 where again each user through his own sharingpolicy my receive a customized feedback from the KDP system 195regarding his own tailored and shared content as well as the originalshared content from user A 101.

While embodiments, implementations, and applications have been shown anddescribed, it would be apparent to those skilled in the art having thebenefit of this disclosure that many more modifications than mentionedabove are possible without departing from the inventive conceptsdisclosed herein.

I claim:
 1. In a computer system having a memory, a processor, and a communication network and configured to receive input from a plurality of users, a method for generating augmented content, the method comprising: displaying at least a first portion of the augmented content using a first display device, wherein the first portion of the augmented content is associated with a first user of the plurality of users; storing a first data information associated with the first user of the plurality of users in the memory; storing a second data information associated with a second user of the plurality of users in the memory; generating the first portion of the augmented content based on the first data information; generating a second portion of the augmented content based on the first portion of the augmented content and the second data information; and displaying the second portion of the augmented content and at least a part of the first portion of the augmented content using a second display device.
 2. The method of claim 1, wherein the computer system is configured to be coupled to the first user of the plurality of users via a first access channel of the communication network.
 3. The method of claim 2, wherein the computer system is further configured to be coupled to a second user of the plurality of users via a second access channel of the communication network.
 4. The method of claim 1, wherein the computer system is configured to be coupled to the first user of the plurality of users and to the second user of the plurality of users via the communication network.
 5. The method of claim 1, wherein, the computer system is operable to have independent access to the memory via a memory channel and to a storage medium configured to store the augmented content
 6. The method of claim 1, wherein the second portion of the augmented content is generated in real-time.
 7. The method of claim 1, further comprising: annotating the augmented content using the first data information associated with the first user.
 8. The method of claim 7, further comprising: sharing the annotated augmented content with one or more users of the plurality of users using a first sharing policy.
 9. The method of claim 8, further comprising: modifying the annotated augmented content based on the second data information associated with the second user of the plurality of users.
 10. The method of claim 8, further comprising: storing a respective data information for each respective user of the plurality of users in the memory; and broadcasting the annotated augmented content to each respective user of the plurality of users based on the respective data information for each respective user of the plurality of users stored in the memory.
 11. A knowledge discovery system configured to receive input from a plurality of users, the knowledge discovery system comprising: a first computer system having a memory, a processor, and a communication network, the processor is coupled to a memory via a memory channel, the memory is configured (i) to store first data information associated with a first user of the plurality of users, and (ii) to store second data information associated with a second user of the plurality of users, the first computer system is operable to have independent access to the memory via the memory channel and to a storage medium configured to store first augmented data associated with the first user of the plurality of users, wherein the first computer system is configured to be coupled (i) to the first user of the plurality of users via a first access channel, and (ii) to a second user of the plurality of users via a second access channel, wherein the first computer system is configured to generate the first augmented data using the first data information, and wherein the first computer system is configured to generate second augmented data by using the second data information to modify the first augmented data.
 12. The knowledge discovery system of claim 11, further comprising: a first display device configured to display at least a portion of the first augmented data.
 13. The knowledge discovery system of claim 12, further comprising: a second display device configured to display at least a portion of the second augmented data.
 14. The knowledge discovery system of claim 11, wherein the computer system is configured to broadcast the second augmented data based on the second data information.
 15. The knowledge discovery system of claim 11, further comprising: a respective sharing policy for each respective user of the plurality of users, wherein the first computer system is configured to generate a respective augmented data for a respective user of the plurality of users by using the respective sharing policy of the respective user of the plurality of users to modify the first augmented data.
 16. The knowledge discovery system of claim 15, wherein the first user of the plurality of users and the second user of the plurality of users are registered users of the knowledge discovery system.
 17. The knowledge discovery system of claim 16, wherein the computer system is further configured to collect third data information associated with a third user of the plurality of users, and wherein the third user is not a registered user of the knowledge discovery system.
 18. The knowledge discovery system of claim 17, wherein the first computer system is configured to generate third augmented data by using the third data information to modify the first augmented data.
 19. The knowledge discovery system of claim 15, wherein the first computer system is configured to restrict generating the respective augmented data for a respective user of the plurality of users if the respective user is not a registered user of the knowledge discovery system.
 20. The knowledge discovery system of claim 15, further comprising: a second computer system coupled to the storage medium, wherein the second computer system is configured to generate a respective augmented data for a respective user of the plurality of users by using the respective sharing policy of the respective user of the plurality of users to modify the first augmented data. 